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ABSTRACT 

Information about the performance of higher education 
graduates and how we can develop better performance indicators is 
limited. There is some information on aspects of the results of 
higher education from synthesis of small scale research projects, 
from standardized tests, and from national surveys. There are also 
efforts underway to design an indicator system that would track 
progress with regard to cognitive outcomes and literacy. A group 
appointed by the National Education Goals Panel has recommended that 
a new assessment system be created for higher education modeled on 
the existing one for elementary and secondary education, the National 
Assessment of Educational Progress. Some preliminary work has been 
done in identifying indicators of good practice. The question is now 
whether efforts *"o secure better information can reach consensus in 
the higher education community. An important element of any indicator 
system must be the academic performance of college graduates in some 
set of content and skill disciplines. Consensus around skills, 
centering on communications, critical thinking, and problem solving, 
is more likely than consensus on content knowledge. A number of 
example tasks are presented as potential measurement techniques. 
Twenty-three figures illustrate the discussion. (SLD) 
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IN BRIEF . . . 



America's extensive and varied 
system of higher education is respected 
around the world. Indeed, it attracts 
students from around the world. 

But pressures have been building on 
higher education over the last decade. 
Concerns about global competitiveness. 
Pinched public budgets. A general demand 
for education reform. Galloping tuition 
costs. The result: higher education is 
increasingly being asked to show what 
results it achieves — what students, 
parents, and the public are getting for 
their money. Quantity has certainly been, 
achieved, but where do we stand on quality? 

Do we have an information system that 
would answer that question? No, we 
certainly do not. This report is about the 
need for such a system, and about recent, 
though incomplete, work to bring it into 
existence. 

Do we know nothing about the out- 
comes of a college education in America? 
No, that's not true either. However, the 
data available are limited, and were not 
designed or gathered specifically to be 
indicators of quality. We present what is 
known, but it adds up to a fragmentary 
and blurred picture. 

The picture for four-year college gradu- 
ates who go on to graduate school looks 
pretty good, in terms of recent trends. We 
know this from the results of the Graduate 
Record Examination (GRE). Quantitative 
and Analytical scores have risen over the 
last dozen years, even as the numbers 
going on to graduate school have risen 
sharply. Verbal scores remained basically 
level. In subject tests in their majors, 



graduates' average scores rose in eight 
subjects and fell in seven. 

There have been many small-scale 
research studies of the outcomes of college, 
and a summary of 2,600 of them shows 
among other things that college has a 
positive effect on verbal and quantitative 
skills, oral and written communication, 
and critical thinking. We would be shocked 
if it were otherwise. This summary doesn't 
tell us what the trends are for such results. 

How literate are college graduates? A 
national household assessment of adults 
in 1992 tells us a whole lot about this, 
using print materials dealing with prose, 
document, and quantitative tasks. They 
o.re certainly more literate, on average, 
than those who do not go to college, or do 
not graduate. But their levels of literate- 
ness range from a lot less than impressive 
to mediocre to near alarming, depending 
on who is making the judgment. In the 
final analysis, it is the reader who must 
make that judgment; there are no estab- 
lished standards of expectation here. We 
suggest these data need to be examined 
carefully, and the resulting message 
debated. 

There are five levels of prose literacy; 
the fifth level is the highest. The bulk of 
four-year and two-year college graduates 
perform at Levels 3 and 4. 

• The 35 percent of four-year gradu- 
ates (and 41 percent of two-year 
graduates) in Level 3 can consis- 
tently do tasks such as writing a 
brief letter to explain a billing error 
or read a news article and identify a 
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sentence that provides an interpre- 
tation of a situation/ 

• The 42 percent of four-year gradu- 
ates (and 34 percent of two year 
graduates) at Level 4 can consis- 
tently perform tasks such as stating 
in writing an argument made in a 
lengthy newspaper article or con- 
trasting the views expressed in two 
editorials on technologies available 
to make fuel-efficient cars. 

• The fact remains that 11 percent of 
four-year graduates (and 21 percent 
of two-year graduates) do not reach 
Level 3 or 4. Most of these gradu- 
ates are in Level 2, where they 
succeed at such tasks as interpret- 
ing the instructions from an appli- 
ance warranty or locating two fea- 
tures of information in a sports 
article. Among young four-year 
graduates, 8 percent are below 
Level 3. 

• Relatively few reach the highest 
level, Level 5. Eleven percent of 
four-year graduates (and 4 percent 
of two-year graduates) are at this 
level, where sample tasks include 
summarizing two ways lawyers may 
challenge prospective jurors, and 
comparing the approaches stated in 
a narrative on growing up. 

• Almost half — 47 percent — of four- 
year graduates (and 62 percent of 
two-year graduates) do not reach 
Levels 4 and 5. These levels are 
described in the report and com- 
plete examples of the tasks are 
provided (as they are for the other 
levels, as well). 

'Percentages are for adults born in the U.S. and its 
territories. 



Similar information is provided for 
quantitative tasks and tasks using docu- 
ments. Over half of four-year graduates 
are below Levels 4 and 5 in quantitative 
proficiency and just under half in docu- 
ment proficiency. A representative quanti- 
tative task at Level 4 is using a pamphlet 
on eligibility to calculate the yearly 
amount a couple would receive for basic 
supplemental security income. A represen- 
tative document task at this level is using 
a bus schedule to determine the appropri- 
ate bus to take for a given set of conditions. 

There is a broad distribution of profi- 
ciency and achievement in the United 
States at any level of academic creden- 
tials, or any grade level in school, and 
these distributions overlap considerably. 
So it may not be terribly informative to 
talk about what all four-year graduates 
can do, as if there were something stan- 
dard about what a college degree means in 
this nation. The top 25 percent of high 
school graduates do better than the bot- 
tom 25 percent of four-year college gradu- 
ates. But a graduate from what kind of 
college, from what kind of high school, and 
from what kind of socioeconomic back- 
ground? A monitoring system needs to 
answer these questions to be used and to 
be useful. 

For college graduates, the levels of 
literateness they achieve are directly 
related to their success in the labor mar- 
ket. The higher the average levels of lit- 
eracy, the higher the average weekly 
wages, and the greater the average num- 
ber of weeks worked during the year. 

Periodically, information is available 
about the outcomes of college in terms of 
the labor market status of four-year 
graduates a year after they leave school. 
Average salaries have been fairly flat 
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since 1976. Of course, they vary consider- 
ably by major. There is also considerable 
variation by major in whether degree 
recipients are working in their fields, in 
whether they have jobs that require a 
four-year degree, and in whether they 
enroll in further education. We summarize 
this information in the report. We think 
these data, produced by the National 
Center for Education Statistics, should be 
more extensively available, and in forms 
students and counselors could use to 
inform choices about fields of study. 

Do we need better, more comprehen- 
sive, information about the outcomes of 
higher education? We believe that we do. 
There is a concern about quality and 
whether it is going up or going down. The 
data now available are not equal to the 
task of measuring levels and trends. 

The National Education Goals, 
announced by President Bush and all the 
governors in 1989, provide a clear starting 
point. They called for a substantial increase 
in the proportion of college graduates "who 
demonstrate an advanced ability to think 
critically, communicate effectively, and 
solve problems." The Goals Panel recom- 
mended a sample-based, national assess- 
ment system to measure progress toward 
this goal, a system similar to what we 
have had for a quarter century for elemen- 
tary and secondary education, the National 
Assessment of Educational Progress. 

Considerable preliminary work has 
been undertaken in the extensive work- 
shops held by the National Center for 
Education Statistics on how best to pro- 
ceed. But moving further has not been 
possible, as additional funds for imple- 
mentation have not been available. Even 
the most modest improvement that such a 
monitoring system would make possible 
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would provide returns many times over 
the resources required to create it. 

Are there risks and shortcomings in 
developing and implementing such a 
monitoring system? Yes, there are, and 
care would have to be taken to minimize 
them. The process would have to achieve 
consensus in the higher education commu- 
nity on what and how to measure. The 
timing is good, though, for there has 
recently been considerable work on assess- 
ment technology, in terms of performance 
assessment and the use of portfolios, for 
example. While a monitoring system 
would be based on samples, and not pro- 
vide ^formation for individual students 
and institutions, it would greatly aid in 
the development of assessment instru- 
ments in higher education generally. The 
knowledge from work on technology and 
measurement approaches can be trans- 
ferred, and information would be available 
for the nation as a whole that could be 
used for comparison. 

But we should keep in mind that not all 
the outcomes of higher education can be so 
precisely quantified. It has been said that 
not everything that counts can be counted, 
and not everything that is counted, counts 
(attributed to Albert Einstein). A parallel 
approach should be pursued, in developing 
"indicators of good practice" that can be 
agreed upon and monitored. We report on 
the work that has proceeded thus far in 
developing such indicators, as well. 

In short, how much learning takes place 
when students proceed toward their degrees, 
what degrees of learning are we achieving in 
our vast education system, and how is this 
changing over time? 
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SUMMARY AND HIGHLIGHTS 



This report is about the limited infor- 
mation we do have about the performance 
of higher education graduates, and how 
we can develop better performance indica- 
tors. While it is generally acknowledged 
that the United States has the best system 
of higher education in the world, colleges 
and universities have come under increased 
pressure to show what they accomplish, as 
public budgets become tighter, as tuition 
becomes more expensive, as concerns 
about global economic competitiveness 
abroad grow, and as the education reform 
movement subjects all aspects of educa- 
tion to scrutiny The governors, the Presi- 
dent, and the U.S. Congress have adopted 
as a goal for the year 2000 that "The 
proportion of college graduates who dem- 
onstrate an advanced ability to think 
critically, communicate effectively, and 
solve problems will increase substantially" 

There is limited information on some 
aspects of the results of higher education 
from syntheses of small scale research 
projects, from the Graduate Record 
Examinations (GRE), from the Graduate 
Management Admissions Test (GMAT), 
from the 1992 National Assessment of 
Adult Literacy, and from periodic surveys 
of the employment and educational status 
of bachelor's degree recipients one year 
after graduation. There are also efforts 
under way to try to design an indicator 
system that would track progress, particu- 
larly toward the national goal set for the 
year 2000. In this report we are dealing 
with general indicators of how well the 
system, and segments of it, are perform- 
ing. Many individual states are creating 
assessments that provide data for specific 
colleges and universities, for purposes of 



accountability and program improvement. 
These are not the focus of this report, 
although these matters are obviously 
related. 

* Cognitive and Other Outcomes 

* In syntheses of individual research 
studies, the collegiate experience has 
been found to have a positive impact on 
verbal and quantitative skills, oral and 
written communication, critical think- 
ing, use of reason and evidence to 
address poorly structured problems, 
and intellectual flexibility. This positive 
impact exists after controlling for a 
variety of student characteristics and 
normal maturation. 

* Quantitative and Analytical scores rose 
between 1991 and 1993 on the General 
Test of the GRE, and Verbal scores were 
stable; at the same time the volume of 
examinees rose 47 percent. 

* Mean scores rose from 1981 to 1993 in 
eight GRE Subject Tests; they fell in 
seven Subject Tests. Seven Subject Tests 
had increases in the number of exam- 
inees, and eight had declines. 

* GMAT scores have improved over the 
last dozen years, with volumes of test 
takers rising until 1990, and then 
dropping sharply (as volume dropped, 
scores held steady). 

* Literateness 

In 1992, ETS, under contract with the 
National Center for Education Statistics, 
assessed a national sample of all adults, 



age 16 and over. From this assessment we 
can determine how literate college gradu- 
ates are, using prose, document, and 
quantitative materials. 

* Prose Literacy - U.S. born adults age 16 
and over 

-Three percent of two-year college 
graduates and 2 percent of four-year 
college graduates performed only at 
Level 1, the lowest level. They can 
locate a single piece of information in a 
text, when there is little distracting 
information to deal with. 

-Eighteen percent of two-year college 
graduates and 9 percent of four-year 
graduates are at Level 2. They can 
locate a single piece of information 
when there is distracting information, 
and they can integrate, compare, and 
contrast information. 

-Forty-one percent of two-year college 
graduates and 35 percent of four-year 
graduates are at Level 3, where they 
can match information in a text to that 
in a directive given to them, when 
some inference is required. 

-Thirty-four percent of two-year college 
graduates and 42 percent of four-year 
graduates are at Level 4, where they 
can integrate and synthesize informa- 
tion from complex or lengthy passages 
and make more complex inferences. 

-Just 4 percent of two-year college 
graduates, and 11 percent of four-year 
graduates are at Level 5. where they 
can make high level inferences based 
on text and can find information in 
dense text with considerable distract- 
ing information that might seem plau- 
sible, but is incorrect. 



* Document Literacy - U.S. born adults age 
16 and over 

-Five percent of two-year college gradu- 
ates and 3 percent of four-year gradu- 
ates are only at Level 1, the lowest 
level. They can match information and 
fill in personal information on forms. 

-Twenty-two percent of two-year col- 
lege graduates and 13 percent of four- 
year graduates are at Level 2. They can 
match information in documents that 
contain distractors, and they can inte- 
grate information from several places 
in a document. 

-Forty-three percent of two-year college 
graduates and 37 percent of four-year 
college graduates are at Level 3. They 
can integrate several pieces of informa- 
tion and deal with rather complex 
tables and graphs that contain infor- 
mation not pertaining to the task. 

-Twenty-six percent of two-year college 
graduates and 39 percent of four-year 
college graduates are at Level 4. They 
can perform tasks requiring greater 
inferences and involving more detailed 
information. 

-Just 4 percent of two-year college 
graduates and 8 percent of four-year 
graduates reach Level 5. They can use 
complex documents that contain 
distracting information and make 
high-level inferences. 

:,: Quantitative Literacy - U.S. born adults 
age 16 and over 

-Four percent of two-year college 
graduates and 2 percent of four-year 
graduates are only at Level 1, the 
lowest level. They can perform single, 
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relatively simple operations (addition 
subtraction, multiplication). 

-Seventeen percent of two-year college 
graduates and 10 percent of four-year 
graduates are at Level 2. They can 
perform an arithmetic operation using 
numbers given to them or easily 
located in the material. 

-Forty-four percent of two-year college 
graduates and 40 percent of four-year 
graduates are at Level 3. They can 
solve problems with two or more num- 
bers located in printed material. 

-Thirty percent of two-yecr college 
graduates and 40 percent of four-year 
graduates are at Level 4. They can 
handle two or more arithmetic opera- 
tions in sequence. 

-Just 5 percent of two-year college 
graduates and 13 percent of four-year 
graduates are at Level 5. They can 
perform multiple arithmetic operations 
sequentially. 

* Levels of Literacy in Relation to Educa- 
tion. Prose proficiency rises with educa- 
tion. Three-fourths of adults with 
between zero and eight years of school 
are in the lowest Level, 1, as are 40 
percent of high school dropouts. Over 75 
percent of two-year college graduates 
and 85 percent of four-year graduates 
reach Level 3. However, just 4 and 10 
percent, respectively, reach Level 5. 

* The Distribution of Literacy by Educa- 
tion. Proficiency varies hugely at each 
level of educational achievement, and 
the distubutions overlap considerably. 
For example, the top 25 percent of high 
school graduates scores higher than the 



bottom 25 percent of four-year college 
graduates. 

* Literacy in Relation to Weeks Worked. 
College graduates work more weeks 
during the year than people with less 
education. Among college graduates, 
those with higher literacy levels work 
more weeks than the others. 

* Literacy in Relation to Weekly Wages. 
Four-year college graduates have higher 
weekly wages than two-year graduates, 
who have higher wages than high school 
graduates. Within all three groups, 
weekly wages are higher at higher 
literacy levels. 

* Literacy in Relation to Age. While there 
is little difference in the literacy profi- 
ciencies of graduates ages 16 to 24 and 
those 25 to 54, graduates who are 55 and 
over have substantially lower literacy 
levels. Whether with two- or four-year 
degrees, older college graduates have 
lower literacy levels than younger ones, 
and average age is generally lower at 
each higher increment in literacy level. 

* Literacy in Relation to the Generations. 
Parents' education influences the 
literacy proficiency levels reached by 
their children. Even among children 
with similar levels of education, those 
with better educated parents have 
higher prose literacy proficiencies. The 
highest preforming group were those 
with four-year college degrees who had 
parents with four-year college degrees. 

* After Graduation 

* A year after graduation, about three- 
fourths of employed graduates with a 
bachelor's degree are in jobs related to 
their fields of study, unchanged since 
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1985. However, 44 percent are in jobs not 
requiring a four-year degree, up from 
1985. 

* Recent degree holders in business 
management and engineering are most 
likely to be employed full-time one year 
after graduation; biology degree hold- 
ers are least likely. 

* Graduates with health, engineering, 
and education degrees are most likely 
to be employed in jobs related to their 
fields of study. Those in humanities 
and social science, public affairs and 
social services are the most likely to be 
working in jobs that do not require a 
four-vear degree. 



science had the largest percentage 
salary increase over 15 years. 

* Health and education majors are the 
most likely to be employed in their 
field of study one year after gradua- 
tion. 

> Toward Higher Education 
Indicators 

* A national goal for the year 2000 is to 
"increase substantially the proportion 
of college graduates who demonstrate 
an advanced ability to think critically, 
communicate effectively, and solve 
problems." How is progress to be moni- 
tored? 



The percent of recent graduates who 
are unemployed a year after gradua- 
tion has varied with the business cycle. 
More have been continuing their 
schooling since the mid 1980s. 

Recipients of bachelor's degrees in 
biology are most likely to continue 
their education and to enroll in gradu- 
ate degree pro; rams; business and 
management degree holders are the 
least likely 

Annua] salaries (in constant dollars) of 
bachelor's degree recipients, one year 
after graduation, have been fairly flat 
since 1976. Average female salaries are 
87 percent of average male salaries, 
ranging from 83 percent in social 
sciences to ] 05 percent in history. 

Bachelor's degree graduates with 
health profession degrees had the 
highest salaries one year after gradua- 
tion; those with degrees in physical 
sciences, mathematics, and computer 



* A group appointed by the National 
Education Goals Panel recommended 
that a new assessment system be 
created for higher education modeled 
upon the existing one for elementary 
and secondary education, the National 
Assessment of Educational Progress. It 
would be an assessment of a represen- 
tative sample of colleges and universi- 
ties to produce a national indicator. It 
also recommended identifying a set of 
"good practice" indicators, practices 
that research shows are tied to 
improved student performance. 

* The National Center for Education 
Statistics issued an RFP to take the 
initial steps to create an assessment 
system. But it was withdrawn due to 
lack of funds. 

* Subsequently, Peter Ewell and his 
associates at the National Center for 
Higher Education Management Sys- 
tems, under contract with the National 
Center for Education Statistics, identi- 
fied indicators of "good practice," where 
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data could be collected through exist- 
ing and new surveys. Also, the Commu- 
nity College Roundtable has identified 
indicators of effectiveness of commu- 
nity colleges. 

With some preliminary work now in 
place, the question is whether the 
resolve exists to move forward to . 
secure better information to help in 
guiding improvement in the higher 
education system. Any such efforts 
would have to strive to reach consensus 
in the higher education community, 
and recognize that the diverse pur- 
poses of higher education cannot all be 
captured in test scores and statistics. 



An important element of any indicator 
system must be the academic perfor- 
mance of college graduates in some set 
of content and skill disciplines. There 
seems to be general consensus around 
skills, such as those included in the 
National Education Goals for the year 
2000, the skills of communications, 
critical thinking, and problem solving. 
The diversity of college and university 
disciplines makes the challenge of 
measuring content knowledge more 
complex. 



ERLC 



13 



.9 



INTRODUCTION 



America's higher education system is 
the envy of the world. It is noted for its 
accessibility, for its dazzling variety, and 
for the high quality of many of its 
programs. 

In most countries, the higher education 
community prides itself on tradition and 
on its role as a conservatory of the nation's 
culture and wisdom. American universi- 
ties and community colleges have been 
unusually flexible and adaptive in meeting 
the rapidly changing needs of our agricul- 
tural, industrial, and technological societ- 
ies. They have adapted while still keeping 
alive some aspects of the European higher 
education tradition. 

This large and vigorous education 
system has been vital to national economic 
growth. It has provided our democratic 
institutions with educated, articulate, and 
informed leaders and constituencies that 
can understand and respond to complex 
issues. As Malcolm Muggridge put it, 
"Higher Education is booming in the 
United States; the Gross National Mind is 
mounting along with the Gross National 
Product." Or, it could likely be put the 
other way around. 

The system is, in fact, too important to 
all of us, and to each individual who goes 
through it, not to be scrutinized. And 
while American higher education retains 
considerable respect at a time when it has 
declined for many other institutions in 
society, it is also feeling pressure from 
converging sources to prove its perfor- 
mance. Among them: 



• A heightened concern for our interna- 
tional economic competitive position 

• Galloping tuition costs, greatly exceed- 
ing the general rate of inflation 

• Pinched public budgets, actual cutbacks 
in funding for many public institutions, 
and demands from many state legisla- 
tures that the higher education product 
be measured 

• A general clamor for education reform, 
now at least a decade old 

• A demand from accrediting agencies 
that colleges and universities show 
evidence that students are meeting 
educational goals 

How these forces affect the views of 
individual citizens is hard to determine. 
Parents worry about their children going to 
college, and what they get for their money. 
They hear of the drinking reputations of a 
lot of student bodies. They read of grade 
inflation, even at schools like Stanford. 
They wonder what polarizing debates 
about such things as political correctness 
mean for the bottom line of learning. In 
any case, it is these individuals who made 
a book by Allan Bloom, The Closing of the 
American Mind: How Higher Education 
Failed Democracy and Impoverished the 
Soul of Today's Students, into a bestseller. 

And in a recent report by the "Wing- 
spread Group on Higher Education," 
chaired by William E. Brock, the group 
charged that a "dangerous mismatch 
exists between what American society 
needs of higher education and what it is 
receiving. No where is the mismatch more 
dangerous than in the quality of under- 
graduate preparation provided on many 
campuses." 
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These conditions and concerns have 
been expressed, among other ways, in the 
accountability legislation at the state 
level. Recently, Missouri and South 
Dakota started senior level testing pro- 
grams. Arkansas joined Florida and Texas 
in having "rising-junior" exams. California 
has imposed outcomes testing on commu- 
nity colleges, and Colorado institutions are 
expected to do "value added" testing from 
the freshman to the senior years. Legisla- 
tion is pending in Texas. 

And in 1989, at the Education Summit 
in Charlottesville, Virginia, President 
Bush and the nation's governors set na- 
tional goals for the entire education sys- 
tem, including higher education, that by 
the year 2000, "The proportion of 
college graduates who demonstrate an 
advanced ability to think critically, com- 
municate effectively and solve problems 
will increase substantially." This led to 
exploring approaches to monitoring 
whether this goal is being achieved. A 
report on these efforts, and what came out 
of them so far, is provided in the section of 
this report entitled "Toward Higher Edu- 
cation Indicators?" 



* * * * 

This report is about the limited quanti- 
tative information now available about the 
results of higher education, and about how 
we might get better information. It's focus 
is on indicators for the nation, or for a 
state, or for a segment of higher education. 
It is not about an accountability system to 
track the performance of a particular 
institution. At the same time, indicator 
systems can play a constructive role in 
informing policy and in serving as a model 
and laboratory for the development of an 
assessment technology that can be applied 



as well at the level of individual institu- 
tions. All this presumes that a process 
would be used to develop an indicator 
system that would achieve broad consensus 
in the higher education community and 
among policymakers responsible for the 
system. 

This broad consensus, assuming it can 
be achieved, would provide some assur- 
ance of relevance to higher education's 
aims and mission. But that consensus 
needs to recognize both the limitations of 
quantitative measurement as well as its 
uses. For no quantitative measurement 
v/ill reflect the accomplishment of all that 
we expect from higher education. Many 
purposes have been articulated for higher 
education, and individual institutions 
would give different emphasis and weights 
to these purposes. How, for example, 
would we determine if we were achieving 
the purpose described by the late Stephen 
K. Bailey, eminent educator and political 
scientist, who said: "I get an education so 
that later in life (rapping his knuckles on 
his forehead), when I knock on me, some- 
body answers." No, we don't think that the 
destiny of higher education should be 
turned over to bean counters. But good 
information, carefully developed, and 
wisely used by minds that can answer 
when somebody knocks, can be construc- 
tive. 

Having established some context for 
the reader to understand the authors' 
perspective, we describe the contents tn 
follow. We have accumulated the data that 
are now available about higher education 
outcomes. These data provide a useful, but 
limited, perspective on the performance of 
graduates of higher education. In thinking 
about what we need, it helps to know what 
we now have. 
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There have been a large number of 
small-scale research studies, synthe- 
sized in a monumental work by Ernest 
Pascarella and Patrick Terenzini. 

The Graduate Record Examination, 
general and subject matter tests, is 
taken by the majority of college gradu- 
ates who go on to graduate education. 
We also include the results of the 
Graduate Management Admissions 
Test, taken by tens of thousands of 
college graduates who go on to busi- 
ness schools. 

There has been a large-scale literacy 
study that sheds light on the literate- 
ness of college graduates. The recently 
administered National Adult Literacy 
Survey (1992) gives us a representative 
set of data in that it was administered 
to a random stratified sample of over 
25,000 adults aged 16 and older in 
households and prisons all across the 
country. Since the samples included 
significant numbers of graduates of 
two- and four-year institutions, we can 
examine in some interesting detail how 
these subgroups compare to other 
groups in the population in terms of 
knowledge, skills, job success, earning 
power, and unemployment histories. 
Many of the assessment tasks attempt 



to measure problem-solving and com- 
munication skills. We can also see how 
college graduates with varying literacy 
levels do in these areas. This consti- 
tutes the largest section of the report. 

• The labor market outcomes for four- 
year college graduates, one year after 
they received their bachelor's degree. 
These data go back to 1976. 

As noted earlier, these data give a 
limited picture of the results of the higher 
education enterprise. Graduate Record 
Examinations are taken only by those 
planning to continue their educations. 
Raising literacy levels may not be consid- 
ered an explicit goal of colleges and uni- 
versities, and while we tout college as the 
way to get ahead in the economy, economic 
success is not the only purpose of higher 
education. We make no pronouncements 
about the overall health of the higher 
education system based on these statistics, 
although we should take from them what 
they do have to offer. 

Finally, we report on recent efforts to 
create a more comprehensive system, in 
the section entitled "Toward Higher 
Education Indicators," and we close with 
some observations about the data provided 
and prospects for the future. 
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COGNITIVE AND OTHER OUTCOMES 



There are, of course, no comprehensive 
outcome measures for college education in 
the U.S., no national examination at 
graduation, and no sample-based assess- 
ment that would permit monitoring what 
happens to students in two or four years 
at college. There have, however, been 
many research studies with specific col- 
lege populations, and there do exist tests 
taken by a large proportion of four-year 
college graduates who apply for graduate 
school. In this brief section, we summarize 
what is known from these sources: 



1. Over 2,600 research studies on the 
effect of college on students have 
been synthesized in a 900-page book 
by Ernest Pascarella and Patrick 
Terenzini. The results are presented 
in summarized form. 

2. The results of the Graduate Record 
Examination (GRE) tests of verbal, 
quantitative, and analytical skills 
for the years 1981-1993 are 
displayed. 

3. The results of the GRE subject 
matter tests are provided, showing 
change since 1981. 

4. The results of the Graduate Man- 
agement Admissions Test (GMAT) 
from 1982 to 1993. 
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Synthesis of 
Research on 
College 
Outcomes 



In the 900-page book, How College Affects 
Students, Ernest Pascarella and Patrick 
Terenzini take on the monumental task 
of synthesizing over 2,600 research 
studies on the effects of college. On the 
page to the right, we summarize the 
results in areas of cognitive develop- 
ment, in terms of the net effect, discount- 
ing for "normal maturation, mere aging, 
or other noncollegiate sources of change." 
These studies also control for a variety of 
student characteristics that vary from 
study to study. 

There are positive impacts in all areas of 
cognitive development studied. 

Pascarella and Terenzini also address 
other areas of development. They find 
moderate or strong evidence for a posi- 
tive effect on: 

• Aesthetic, cultural, and intellectual 
values 

• Value placed on a liberal education 

• Value placed on intrinsic occupational 
rewards 

• Political liberalism 

• Gender roles ("toward the v modern'") 

• Academic self-concept 

• Social self-concept 

• Self-esteem 

• Intellectual orientation 

• Personal adjustment and psychological 
well-being 

• Use of principled reasoning in judging 
moral issues 



Source: Ernest T, 
Pascarella and 
Patrick T. 
Terenzini, How 
College Effects 
Students, Jossey- 
Bass, 199L 



18 



16 



College has been found to have a positive impact on verbal and 
quantitative skills, oral and written communication, critical 

thinking, use or reason and evidence to address ill-structured 
problems, and intellectual flexibility, after controlling for a variety 

of other characteristics, and such factors as normal maturation. 



Synthesis of Research Studies on the Net 1 
Effects of College on Learning and Cognitive Development 

! 

j • General Verbal Skills 

I Strong evidence of a positive effect, after controlling for precollege verbal skills, race, and socioeconomic 

I status. Graduates have a 10 to 13 percentile point advantage, over those not going to college. 



I • General Quantitative Skills 

I Strong evidence of a positive effect, after controlling for precollege quantitative skills, race, and socioeco- 
nomic status. Graduates have a 11 to 13 percentile point advantage. 

• Oral Communication Skills 

Moderate evidence of a positive effect, after controlling for age and academic ability. The magnitude of the 
net effect is unclear. 2 

• Written Communication Skills 

Moderate evidence of a positive effect, after holding age and academic ability constant. The magnitude of 
the net effect is unclear. 2 

• General Intellectual and Analytical Skill Development 

Moderate to strong evidence of a positive effect, after controlling for age, verbal ability, and quantitative 
ability. Magnitude of the effect is unclear. 2 

! 
I 
I 

! • Critical Thinking 

i Strong evidence of a positive effect, after controlling for precollege critical thinking, academic aptitude, 
i socioeconomic status, and educational aspirations. Freshman year net effect was 17 percentile points. 
Magnitude of the net effect for all four years is unclear. 2 

• Use of reason and evidence to address ill-structured problems (reflective judgment, informal reasoning) 
Moderate to strong evidence of a positive impact, after controlling for age, intelligence, and academic 
aptitude. Magnitude unclear. 2 

& Intellectual flexibility 

I derate to strong evidence of a positive impact, after controlling for age, intelligence, and academic impact. 
• Magnitude unclear. 2 



[ The college's net or unique impact, "as distinct from normal maturation, mere aging, or other noncollegiate sources of 

I change." , , 

1 ^Unclear," as used by the autl.v rs here, means that they acknowledge that the studies do not allow such estimates or 

, ] tha* the evidence, though generally consistent, is still sufficiently complex to make an estimate of effect size h Q zardo us ; 
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Graduate 
Record 
Examinations 
General Test 



Quantitative and Analytical scores on 
the General Test of the Graduate Record 
Examinations (GRE) rose from 1981 to 
1993, while verbal scores remained 
about the same. On a scale of 200 to 800, 
mean scores on the Quantitative mea- 
sure rose 27 points from 1981 to 1993. 
Scores on the Analytical measure rose 
35 points. On the Verbal measure, there 
was only a small decline of 3 points (see 
Figure 1). Scores declined slightly in the 
early 1990s, as tho volume of test takers 
shot up substantially 

Thus scores rose or remained fairly 
constant at a time when the number of 
test takers rose sharply. In 1993, 
199,550 took the examination, an 
increase of 47 percent over 1981. 

By this measure, the scholastic abilities 
of applicants to graduate school have 
been increasing substantially in quanti- 
tative and analytical ability, with little 
change in verbal ability. Typically scores 
on such tests decline as volume rises; it 
is fair to conclude that there were solid 
gains in the skills these tests measure. 



The General Test of 
the Graduate 
Record Examina- 
tions yields 
separate scores for 
the verbal, quanti- 
tative, and analyti- 
cal abilities related 
to success at the 
graduate level of 
education. 

• The verbal 
measure employs 
four types of 
questions: ant- 
onyms, analogies, 
sentence comple- 
tions, and reading 
comprehension. 

• The quantitative 
measure employs 
three types of 
questions: discrete 
quantitative 
questions, data 
interpretation 
questions, and 
quantitative 
comparison 
questions. 

• The analytical 
measure is of 
analytical reason- 
ing and logical 
reasoning. 

Most examinees 
apply to graduate 
school, but not all 
enroll. Not all 
graduate depart- 
ments in the 
United States 
require the GRE 
General Test. Even 
though the number 
of departments 
that use the test is 
large and appears 
to be increasing, 
GRE General Test 
examinees are not 
necessarily repre- 
sentative of all 
applicants to or 
enrollees in a field 
or in graduate 
education gener- 
ally. 
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Quantitative and Analytical scores rose between 1981 and 1993 
on the General Test of the Graduate Record Examination, 
and Verbal scores were stable. At the same time, 
the volume of examinees increased 47 percent. 



Figure 1: GRE Score Trends, 1981 - 1993 
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Note: Data are tor college seniors and non-enrolled college graduates. 
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Graduate 
Record 
Examinations 
Subject Tests 



Large increases on GRE Subject Test 
volumes between 1981 and 1993 occurred 
in computer science, literature, physics, 
mathematics, sociology, history, and 
psychology. Score increases occurred in 
computer science, math, sociology, and 
psychology; decreases occurred in the 
other subjects (see Figure 2). 

Decreases in the volume of test-takers 
occurred in geology, music, education, 
economics, biology, political science, 
chemistry, and engineering. 

Figure 2 also shows GRE Subject Test 
volumes for the 1992-93 testing year. 



Scores on the GRE 
Subject Tests are 
intended to indi- 
cate students' 
knowledge of the 
subject matter 
emphasized in 
many undergradu- 
ate programs as 
preparation for 
graduate study. 
For some Subject 
Tests, subscores 
are provided in 
addition to the 
total score; these 
subscores indicate 
the strengths and 
weaknesses of 
individual students' 
preparation. 

Total testing time 
for each Subject 
Test is 2 hours and 
50 minutes. 
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Mean scores rose from 1981 to 1993 in eight GRE Subject Tests; 

they fell in seven subject tests. 
Seven subject tests had increases in examinees, 
and eight had declines. 



Figure 2: Changes in the Number of Examinees and 
Mean Scores, GRE Subject Tests, 1981 - 1993 
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1993 Test Takers 


Computer Science 


2,832 


Literature 


5,197 


Physics 
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Math 
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Engineering 


3,189 



Note: Data are for college seniors and non-enrolled college graduates. 
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Average GMAT scores have been on the 
upswing over the last decade, increasing 
from 481 in 1982 to 500 in 1993 for U.S. 
test takers. During the same decade, the 
number of U.S. test takers reached an 
all-time high of more than 160,000 in 
1990; rebounding from a low of about 
114,000 in 1984 (see Figures 3 and 4), 
but dropped back sharply to 135,000 by 
1993. 

These test takers represent the vast 
majority of applicants to MBA or Ph.D. 
programs in business and management 
fields. 



The GMAT is a 
test of developed 
abilities that is 
sponsored and 
directed by the 
Graduate Manage- 
ment Association 
Council and 
administered by 
Educat ional 
Testing Service. 
GMAT results 
provide counselors 
with one predictor 
of academic 
performance in 
graduate manage- 
ment school. Scores 
on the GMAT are 
currently used by 
about 1,000 
graduate manage- 
ment programs 
throughout the 
world and are 
required of every 
applicant by about 
780 institutions. 
Thus, this pool of 
test takers 
represents the 
vast majority of 
applicants to MBA 
or Ph.D. programs 
in business and 
management fields. 
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Graduate 
Management 
Admission 
Test (GMAT) 



Graduate Management Admission Test (GMAT) 
scores have increased steadily throughout the decade. 



Figure 3: Trends in Average GMAT Scores 
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Figure 4: Trends in Average GMAT Scores by Race/Ethnicity 



550 



500 




450 



Mean Score 
540 



520 



White 




, , , , , 1 1 1 1 » 1 1 1 1 1 1 11 



500 



480 



460 



440 



420 



400 




380 



360 l-t 



'84 '85 '86 '87 '88 '89 '90 
The descriptor. "non-Hispanic," was attached to the "White" sub-group categorv 
and the additional sub-group "Other Hispanic/Latin American," was added as an 
option in 1992-1993. The 1993 mean score for "Other Hispanic/Latin American" 
i was 461. . _ _ 
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LITERATENESS 



We ask in this section, how literate are 
college graduates? To answe v this question 
we draw upon the results of the first com- 
prehensive study of literacy in the United 
States, a national study carried out in 
nearly 27,000 homes, and representative 
pf all adults age 16 and over living in 
households or federal and state prisons. It 
was mandated by Congress and carried out 
in 1992 by Educational Testing Service 
under contract with the National Center 
for Education Statistics. 

We use the term "literateness" in the 
title to make a point. The survey is of 
"literacy." However, in recent usage, the 
term literacy has come to be thought of 
simply as the opposite of "illiteracy" which 
is understood as being unable to read or 
write at all, being unable to decode the 
printed word, or comprehend it. So we 
tend to think of a literacy study as one 
that finds out how many so-called illiter- 
ates there are. We also remember the 
"literacy test" imposed for voting qualifica- 
tion in some states. Coming at it with this 
mindset would raise the natural question, 
why would such a study have relevance to 
those who complete a higher education? 

Actually, the term "literate" has a very 
broad definition. Its first definition, in 
Webster's Collegiate Dictionary, is "edu- 
cated, cultured," its second is "able to read 
and write," and its third is "versed in 
literature or creative writing." The term 
"illiteracy" includes "a mistake or crudity." 
The National Adult Literacy Study is not a 
survey of "illiteracy." The definition of 
literacy used in this large scale assess- 
ment is as follows: 



Using printed and written informa- 
tion to function in society, to achieve 
one's goals, and to develop one 1 s 
knowledge ana potential. 

The 1992 literacy study was not 
designed to set "cut-points" to count the 
number of illiterates, a term not used in 
the report, Adult Literacy in America, 
written by Irwin Kirsch and his colleagues. 
Rather, it profiles the literateness of the 
entire population, whatever the occupa- 
tional and educational achievements. The 
tasks are centered on real life encounters 
with the printed word, are performance- 
oriented, and have a range of difficulty 
that reaches to, and beyond, those educated 
a^the graduate level. 

Given this definition, and the 165 tasks 
used to operationalize it, we can answer 
the question — how literate are Ameri- 
cans who have college educations, specifi- 
cally those with two-year and four-year 
degrees. While in a general way we expect 
those who are better educated to be more 
literate by such a definition, it must be 
asked; In what way, if any, can we... 
should we... judge the quality of such 
education by how well graduates do on 
these 165 tasks? We know of no college 
curriculum that directly addresses negoti- 
ating these common tasks encountered in 
living. Moreover, we know that higher 
education serves a variety of purposes, 
that institutions follow different paths to 
them, and that there is great debate and 
disagreement over these purposes. Indeed, 
we ask elsewhere in this report whether 
— and with what advantages, what limi- 
tations, and what dangers — any stan- 
dardized as^.i'ssment instrument can 
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measure the degree of achievement in 
higher education. In the final analysis, it 
is a question only the interested and 
thoughtful reader of this report can 
answer. What we will do in the following 
pages is present the results of the assess- 
ment, in a way that should enable the 
reader to form judgements about its utility 
and what it conveys about the literateness 
of higher education graduates. 

But to do this the reader needs to know 
a little about the assessment (a more 
extended, but still short, discussion can be 
found in the ETS Policy Information Cen- 
ter publication, Becoming Literate About 
Literacy). 

The definition of literacy used in this 
assessment, and described above, encom- 
passes the multifaceted nature of literacy 
— it is not a single skill, i.e. reading, but 
rather literacy involves an ordered set of 
skills that are necessary to accomplish 
various tasks in various contexts. Balanc- 
ing a checkbook, obtaining information 
from a train schedule, and understanding 
a passage from a novel — each of these 
tasks calls for literacy skills, but not the 
same skills. The National Adult Literacy 
Survey employed three proficiency scales 
to measure these distinct skills: prose, 
document, and quantitative. Their defini- 
tions are below. 

Prose Literacy - the knowledge and 
skills needed to understand and 
use information from texts that 
include editorials, news stories, 
poems, pnd fiction; for example, 
finding a piece of information in a 
newspaper article, interpreting 
instructions from a warranty, 
inferring a theme from a poem, or 
contrasting views expressed in au 
editorial. 



Document Literacy - the knowledge 
and skills required to locate and 
use information contained in mate- 
rials that include job applications, 
payroll forms, transportation 
schedules, maps, tables, and 
graphs; for example, locating a 
particular intersection on a street 
map, usii-g a schedule to choose the 
appropriate bus, or entering infor- 
mation on an application form. 

Quantitative Literacy - the knowl- 
edge and skills required to apply 
arithmetic operations, either alone 
or sequentially, using numbers 
embedded in printed materials; for 
example, balancing a checkbook, 
figuring out a tip, completing an 
order form, or determining the 
amount of interest from a loan 
advertisement. 

More than 400 trained interviewers 
conducted the survey, administering the 
literacy assessment to over 26,000 indi- 
viduals during the first eight months of 
1992. A separate assessment was carried 
out for the prison population. The assess- 
ment consisted of a set of tasks that simu- 
lated real life situations encountered at 
home, at work, and in the community. 
Almost all of the tasks required partici- 
pants to construct their responses, as 
opposed to choosing a response from mul- 
tiple choices. 

Twelve states also asked ETS to collect 
samples of their populations so they can 
now compare their state results to one 
another and to the national statistics. 

The background questionnaire, which 
took about 20 minutes to administer, 
asked about: 
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• general language background 

• educational background and 
experience 

• political and social participation 

• labor force experience 

• literacy activities 

• demographic information 

The literacy assessment consisted 
almost entirely of simulation tasks. These 
emphasized a broad range of literacy skills 
used in home, work, and in social contexts. 
Many of the tasks required brief oral and 
written responses and asked how prob- 
lems were set up and resolved. Some 
involved the use of a simple, four-function 
calculator. 

One way of presenting the survey 
results would be to describe how all adults 
did on each of the 166 tasks. To do so 
would place a very large burden on the 
reader, making it difficult to conclude 
what the state of literacy is for each popu- 
lation group. To help make such judg- 
ments, statistical methods were used to 
construct proficiency scales from the 



answers to the tasks, scales similar to the 
ones used to report the results of the SAT 
The National Adult Literacy Survey scales 
range from 0 to 500. Individual tasks are 
assigned scores along the scales, to give 
examples of what people with a particular 
score are likely to be able to do. The scales 
were then divided into five levels, each 
encompassing a defined score range, such 
as from 226 to 275. Individuals scoring 
within these scale levels would have a 
high probability of perform! * the tasks at 
the level successfully (a high probability is 
defined as at least 80 percent of the time).* 

In the pages that follow, we show what 
each level of literacy means for each of the 
three literacy scales. For each level, we 
show the percent of the adult population 
at that level, and the percent of two- and 
four-year college graduates at that level, 
separately for those born in the U.S. and 
its territories, and those born elsewhere. 
We also show the relationship of literacy 
to weeks worked, average wages, age, and 
education of parents, for both two- and 
four-year college graduates. 



*And they would have a small chance of performing 
tasks at a higher level. 
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PROSE LITERACY, LEVEL 1 (0 TO 225) 



Three percent of U.S. 
born two-year college 
graduates and 2 percent 
of four-year college 
graduates perform at the 
lowest Prose Level 
(Level 1). They can 
locate a single piece of 
information in a text, 
when there is little 
distracting information 
to deal with. 
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They can: 

read relatively short text to locate a single piece of 
information that is identical to or synonymous with, 
the information given in the question or directive. 
Distracting information is minimal.* 

They are likely to succeed at tasks such as: 

• Identify the country in a short article (149) 

• Locate one piece of information in a sports article 
(210) (See below) 

• Underline a sentence explaining the action stated 
in a short article (225) 

'However, some were at the lower end of this level where 
they could not perform even these routine tasks. 



EXAMPLE TASK 

Underline the sentence (hat tells what Ms. Chanin ate during the stvim 
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Swimmer completes 
Manhattan marathon 



The A**ociated Treat 

NEW YORK— University of Maryland 
senior Stacy Chanin on Wednesday became 
the first person to swim three 28-mile laps 
around Manhattan. 

Chanin, 23, of Virginia, climbed out of 
the East River at 96th Street at 9:30 p.m. 
She began the swim at noon on Tuesday. 

A spokesman for the swimmer, Roy 
Bnmett, said Chanin had kept up her 
strength with "banana and honey" 
sandwiches, hot chocolate, lots of water 
and granola bars." 



Chanin has twice circled Manhattan 
before and trained for the new feat by 
swimming about 28.4 miles a week. The 
Yonkers native has competed as a swimmer 
since she was 15 and hoped to persuade 
Olympic authorities to add a long-distance 
swimming event. 

The Leukemia Society of America 
solicited pledges for each mile she swam. 

In July 1983, Julie Ridge became the 
first person to swim around Manhattan 
twice. With her three laps, Chanin came 
up just short of Diana Nyad's distance 
record, set on a Florida-to-Cuba swim. 
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PROSE LITERACY, LEVEL 2 (226 TO 275) 



Just under one in five 
U.S. born two-year 
college graduates, and 
one in 11 four-year 
graduates function at 
Prose Level 2. They can 
locate a single piece of 
information when there 
is distracting informa- 
tion, and they can 
integrate, compare, and 
contrast information. 
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They can: 

locate a single piece of information when there is 
distracting information or other information that 
seems plausible as the answer but is incorrect. They 
can also integrate, compare, or contrast two or more 
pieces of information. 

They are likely to succeed at tasks sjuch as: 
• Underline meaning of a term given in government 

brochure on supplemental security income (226) 
m Locate two features of information in a sports 

article (250) 

m Interpret instructions from an appliance warranty 
(275) (See below) 



EXAMPLE TASK 



A manufacturing company provides its customers with the fol- 
lowing instructions for returning appliances for service: 



When returning appliance for servicing, include a note telling as clearly and 
as specifically as possible what is wrong with the appliance. 



A repair person for the company receives four appliances with the 
following notes attached. Circle the letter next to the note which 
best follows the instructions supplied by the company. 



The clock does not run 
correctly on this clock 
radio. I tried fixing it, but 
I couldn't. 



The alarm on my clock 
radio doesn't go off at the 
time I set. It rings 15-30 
minutes later. 



My clock radio is not working. It 
stopped working right after 1 
used it for five days. 



This radio is broken. Please 
. repair and return by United 
Parcel Service to the address on 
my slip. 
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PROSE LITERACY, LEVEL 3 (276 TO 325) 



EXAMPI-E TASK 



Two in five U.S. born two- 
year college graduates, and 
over one in three four-year 
graduates can match infor- 
mation in a text to that in 
the directive given to them, 
when some inference is 
required. 



List two things that Chen became involved in or has done to help resolve 
conflicts due to discrimination. 
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They can: 

match information in the 
text and in the task when 
low level inferences are 
required, integrate infor- 
mation from dense or 
lengthy text, and gener- 
ate a response based on 
information easily identi- 
fied in the text. 

They are likely to 
succeed at tasks such 
as: 

> Write a brief letter 
explaining error made 
on a credit card bill 
(288) 

• Read a news article and 
identify a sentence that 
provides interpretation 
of a situation (304) 

• Read a lengthy article to 
identify behaviors that 
meet a stated condition 
(316) 



IDA CHEN is the first Asian-American woman to 
become a judge of the Commonwealth of Pennsylvania. 



She understands 
discrimination because she 
has experienced it herself. 

Soft -spoken and eminently cKgnificd, 
Judge Ida Chen prefers hearing about a 
new acquaintance rather than talking 
about herself. She wants to know about 
career plans, hopes, dreams, fears. She 
gives unsolicited advice as well as 
encouragement. She instills confidence. 

Her father once hoped that she 
would become »? professor. And she 
would have aiso made an outstanding 
social worker or guidance counselor. 
The truth is that Chen wears the caps of 
all these professions as a Family Court 
judge of the Court of Common Pleas of 
Philadelphia County, as a participant in 
public advocacy for minorities, and as a 
particularly sensitive, caring person. 

She understands discrimination 
because she has experienced it herself. 
As an elementary school student, Chen 
trit^ to join the local Brownie troop. 
"You can't be a member," she was told. 
"Only American girls arc in the 
Brownies." 

Originally intent upon a career as a 
journalist, she selected Temple Univer- 
sity because of its outstanding journal- 
ism department and affordable tuition. 
Independence hcing a personal need, she 
paid for her tuition by working for 
Temple's Department of Criminal 
Justice. There she had her first encoun- 
ter with the legal world and it turned 
her career plans in a new direction — 
law school. 

Through meticulous planning, Chen 
was able to earn her undergraduate 
degree in two and a half years and she 
continued to work three jobs. But when 
she began her first semester as a Temple 
law student in the fall of 1973, she was 
barely able to stay awake. Her teacher 
Lynnc Abraham, now a Common Pleas 
Court judge herself, couldn't help hut 
notice Chen yawning in the back of the 
class, and when she determined that 
this student was not a party animal hut 
a workhorse, she arranged a teaching 
assistant's job (or Chen on campus. 

After graduating (torn Temple Law 
School in 1976, Chen worked for the 
U.S. Equal Employment Opportunity 
Commission where she was a litigator 
on behalf of plaintiffs who experienced 
discrimination in the workplace, and 



then moved on to become the first 
Asian-American to serve on the 
Philadelphia Commission on Human 
Relations. 

Appointed by Mayor Wilson Goodc, 
Chen worked with community leaders 
to resolve racial and ethnic tensions and 
also made time to contribute free legal 
counsel to a variety of activist groups. 

The "Help Wanted" section of the 
newspaper contained an entry that 
aroused Chen's curiosity — an ad for a 
judge's position. Her application 
resulted in her selection by a state 
judicial committee to fill a scat in the 
state court. And in July of 1988, she 
officially became a judge of the Court of 
Common Pleas. Running as both a 
Republican and Democratic candidate, 
her position was secured when she won 
her scat on the bench at last Novem- 
ber's election. 

At Family Court, Chen presides over 
criminal and civil cases which include 
adult sex crimes, domestic violence, 
juvenile delinquency, custody, divorce 
and support. Not a pretty picture. 

Chen recalls her first day as judge, 
hearing a juvenile dependency case — 
"It was a horrifying experience. I broke 
down because the cases were so 
depressing," she remembers. 

Outside of the courtroom, Chen has 
made a name for herself in resolving 
interracial conflicts, while glorying in 
her Chinese- American identity. In a 
1986 incident invol ving the desecration 
of Korean street sifns in a Philadelphia 
neighborhood, Clv.n called for a 
meeting with the leaders of that 
community to help resolve the conflict. 

Chen's interest in community 
advocacy is not limited to Asian 
communities. She has been involved in 
Hispanic, Jewish and Black issues, and 
because of her participation in the 
Ethnic Affairs Committee of the Anti- 
Defamation League of B'nai B'rith, 
Chen wa<$ one of 10 women nationwide 
selected to take part in a mist' on to 
Israel. 

With her recently won mandate to 
judicate in the affairs of Pennsylvania's 
citizens, Chen has pledged to work 
tirelessly to defend the rights of its 
people and contribute to the improve- 
ment of human welfare. She would have 
made a fabulous Brownie. 

— Jessica Schultz 
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PROSE LITERACY, LEVEL 4 (326 TO 375) 



One in three U.S. born 
two-year college gradu- 
ates, and over two in five 
four-year graduates can 
integrate and synthesize 
information from complex 
or lengthy passages and 
make more complex 
inferences. 
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They can: 

match text with multiple features, integrate or 
synthesize information from coiuplex or lengthy 
passages, and make more complex inferences. 

They are likely to succeed at tasks such as: 

• State in writing an argument made in a lengthy 
newspaper article (328) 

• Contrast views expressed in two editorials on 
technologies available to make fuel-efficient cars 
(359) (See below) 

• Compare two metaphors used in a poem (374) 



EXAMPLE TASK 



Contrast Dewey 's and Hannas views about the existence of technologies that can be used to 
produce more fuel efficient cars while maintaining the size of the cars. 



Face-Off; Getting More Miles Per Gallon 

Don't demand end 
to cars people want 



Demand cars with 
better gas mileage 



By Robert Dewey 
Gueet columnist 



WASHINGTON — Warning Auto- 
maker* are resurrecting their heavy* 
metal dinosaur*, aka g aa guixlera. 

Government re porta show that average 
new-car mileage haa declined to 28.2 miles 
per gallon — the 1986 level. To revsrse 
this trend, Congreee muit significantly 
increaee existing gas-mileage etandarde. 

Mora than half our Nobel laureate* 
end 700 member* of the National Acad- 
emy of Sciences recently called global 
warming "the moat serious environmen- 
tal threat of the 21et century." In 1989, 
oil import* climbed to a near-recoTd 46% 
of U.S. consumption. Increasing gas 
mileage is the single biggeet step we can 
take to reduce oil import* and curb global 
warming. Greater efficiency also lowers 
our trade deficit (oil imports represent 
40% of it) and decreaeee ths need to drill 
in pristine areas. 

Bigger engines and bigger cars mean 
bigger profits for outomakers, who offer 
ua the products they want us to buy. 
More than ever, American* want prod' 
ucts that have less of an environmental 
impact But with only a few fuel -efficient 
cars to choose from, how do w* find ones 
that meet all our needs? 

Government studiee show aulomakere 
have the technology to dramatically im- 



prove gas mileage — while maintaining 
the 1987 levels of comfort, performance 
and site mix of vehicles. Automakers also 
have the ability to make their products 
aafer. The coat of theas improvement* 
will be offset by savings at the gas pumpl 
Cars can average 45 mpg and light 
trucks 95 mpg primarily by utiliiing en- 
gine and transmission technologies al- 
ready on a few cars today. Further im- 
provements aw poeiible by using tech- 
nologies like the two-stroke engine and 
better aerodynamics that have been de- 
veloped but not used. 

When the current vehicle efficiency 
standards were propoeed in 1974, Ford 
wrongly predicted that they "would re- 
quire either all aub-Pinto-eiaed vehiclee 
or some mix of vehicles rsnging from a 
eub-subcompect to perhaps a Maverick." 
At thet time, Congress required a 100% 
efficiency increase; raising gss mileage 
to 45 mpg requires only a 60% increase. 

Americans want comfortable, safe and 
efficient cars. If automakers wont pro- 
ride them. Congress must mandate them 
when it considers the issue this summer. 

Let* hope lewmakers put the best in- 
terest of the environment and the nation 
shesd of the automakers' lobbyists snd 
political action committees. 



By Thomas H. Hanna 
Guest columnist 



DFTROFT — Do Americans look for- 
ward to the day whan they'll have to haul 
groceries, shuttls ths kids to and from 
school or take family vacations in compact 
and ru boom pact cars? 

T doubt it — which is why U.8. and 
import carmaksTf oppose the 40-milss- 
per-galbn to 45 mpf corporate average 
fuel economy mandates that some are 
pushing in Congraes, either to curb tailpipe 
carbon cUcrifoemieitons because of alleged 
global wanning or for energy conservation. 

Since the mid-1970s, automakers have 
doubled the fleet average fual economy of 
new care to 28 mpc— and further progreaa 
will be mede. 

Compact and subcompact cars with 
mileage of 40 mpg or better are now 
available, yet they appeal to only 6% of 
U.S. car buyers. 

But to schieve a U.S. fleet average of 40 
mpg to 46 mpg. carmaksra would have to 
sharply limit the aveilability of family- 
site models and dramatically trim the size 
end weight of most cars. 

There simply are not magic technolo- 
gies to meet rich a standard. 
Almost every car now sold In the USA 



Roinn Div<y it o conMnrntion annlyti fo* tht iViw 
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would have to be drastically downsiaed, 
and many would be obsolete. 

As a result, Americans each year would 
be una We to buy ths vehicles moat suHad 
for their neade: mid- and family site 
models, luxury automobile*, mini-vans, 
small trucks and utility vehicle*. 

The fleet shift to compacts and suboorn- 
pacte could also fores the closing assem- 
bly plants, auppllsr firms and dealerships, 
at a coat of thousands of U.S. jobs. 

Although a growing number of scientists 
era skeptical of global warming, the issue 
deeervee thorough international scientific 
evaluation, not premature unilateral U.S. 
action. 

Carbon dioxide emissions from U.8. ve- 
hicles trii\ leee than 2.6% of worldwide 
•greenhouse" gases. Even doubling today's 
corporate average fual economy for U.S. 
cars — if technically possible — would cut 
those gasss about .6% 

Whatever the motivation — alleged 
global warming or energy conservation -— 
the stakes are high for millions of Amari* 
cans and thousands of U.S. jobs In unreal- 
istic corporate average fuel economy 
mandates. 

Thorn at H. Hcmm u pnttdtnt mrt <hmf MKfetfM 
•fltttr of tKi Motor VthvU Manufl*tvtri Amocit 
Honcftht Unl*4StQU*. 
RtprvHtd by pirmxtikn of USA Tbdtty. 
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PROSE LITERACY, LEVEL 5 (37G TO 500) 



Just one in 25 U.S. corn 
two-year college graduates 
and one in nine four-year 
graduates can make high 
level inferences based on 
text and can find informa- 
tion in dense text with 
considerable distracting 
information that might 
seem plausible but is 
incorrect. 
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They can: 

find information in a dense 
text that contains consid- 
erable information that is 
distracting — information 
that may seem plausible 
as an answer but is incor- 
rect. Also, they can make 
high-level inferences or 
use specialized background 
knowledge. 

They are likely to 
succeed at tasks suck 
as: 

• Compare the approaches 
stated in a narrative on 
growing up (382) 

• Summarize two ways 
lawyers may challenge 
prospective jurors (410) 

• Interpret a brief phrase 
from a lengthy news 
article (423) 



EXAMPLE TASK 

Identify and summarize the two kinds of challenges that attorneys use 
while selecting members of a jury. 
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DO YOU HAVE A QUESTION? 



QUESTION: What is the new program tor 
scheduling jurors? 

ANSWER: This h a new way of organizing 
art scheduling jurors that is being Intro- 
duced aM over the country. The goals of 
this program are to save money, Increase 
the number of citizens who are summoned 
to serve and decrease the Inconvenience 
of serving. 

The program means that instead of call- 
ing Jurors for two weeks, Jurors now serve 
onfy one day, or for the length of one trial 
Jf they are selected to hear a case. Jurors 
who are not selected to hear a case are 
excused at the end of the day, and their 
obligations to serve as jurors are fuHHted 
for three years. The average trial lasts 
two days once testimony begins. 

An Important part of what is called the 
One Day - One Trial program is the 
"standby" juror. This Is a person ceiled to 
the Courthouse H the number of cases to 
be tried requires more jurors than origi- 
nally estimated. Once called to the Court- 
house, the standby becomes a "regular* 
juror, and Ms or her service Is complete at 
the end of one day or one trial, the same 
as everyone else. 



Q. How was I summoned? 

A The basic source for names of eligible 
jurors is the Driver's License list which is 
supplemented by the voter registration 
1st Names are chosen from these com- 
bined Ksts by a computer m a completely 
random manner. 

Once in the Courthouse, Jurors are 
selected for a trial by this same computer 
and random selection process. 

Q. Mow is the Jury for a particular trial 
selected? 

A When a group of prospective jurors Is 
selected, more than the number needed 
for a trial are called. Once this group has 
been seated In the courtroom, either the 
Judge or the attorneys ask questions. 
This is called voir dire. The purpose of 
questions asked during voir dire Is to 



ensure that aft of the jurors who are 
selected to hear the case will be unbi- 
ased objective and attentive. 



In most cases, prospective jurors wHI be 
asked to raise their hands when a particu- 
lar question applies to them. Examples of 
questions often asked are: Do you know 
the Plaintiff, Defendant or the attorneys in 
this case? Have you been involved in a 
case similar to this one yourself? Where 
the answer is yes, the Jurors raising hands 
may be asked addfrJonat questions, as 
the purpose is to guarantee a fair trial for 
all parties. When an attorney bolieves 
that there Is a legal reason to excuse a 
juror, ho or she wW challenge the juror for 
cause. Unless both attorneys agree that 
the juror should be excused, the Judge 
must either sustain or override the chal- 
lenge. 

After all challenges for cause have been 
ruled upon, the attorneys wHI select the 
trial jury from those who remain by exer- 
cising peremptory challenges. Unlike 
challenges for cause, no reason need be 
given for excusing a juror by peremptory 
challenge. Attorneys usually exercise 
these challenges by taking turns striking 
names from a Hst until both are satisfied 
with the Jurors at the top of the list or until 
they use up the number of challenges 
allowed. Challenged jurors and any extra 
jurors will then be excused and asked to 
return to the jury selection room. 

Jurors should not feel rejected or insulted 
tf they are excused for cause by the Court 
or peremptorily challenged by one of the 
attorneys. The voir dire process and 
challenging of jurors Is simply our Judicial 
system's way of guaranteeing both par- 
ties to a lawsuft a fair trial. 



Q. Am I guaranteed to serve on a Jury? 

A. Not all jurors who are summoned actually 
hear a case. Sometimes all the Judges 
we rid working on trials from the previ- 
ous day, and no new jurors are chosen. 
Normally, however, some new cases begin 
every day. Sometimes Jurors are chal- 
lenged and rat selected. 
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DOCUMENT LITERACY, LEVEL 1 (0 TO 225) 



One in 20 U.S. born two- 
year college graduates, 
and one in 33 four-year 
graduates are at Level 1 
in Document Literacy. 
They can match infor- 
mation and fill in 
personal information on 
forms. 
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They can: 

locate a piece of information based on a literal match 
between the task and the document or enter infor- 
mation from personal knowledge onto a document. 
Little, if any, distracting information is present.* 

They are likely to succeed at tasks such as: 

• Sign their names (69) 

• Locate time of meeting on a form (180) 

• Use a pie graph to locate type of vehicle having 
specific sales (214) 

'However, some were at the lower end of this level where 
they could not perform even these routine tasks. 



EXAMPLE TASK 



You have gone to an employment center for help in finding a 
job. You know that this center handles many different kinds of 
jobs. Also, several of your friends who have applied here have 


found jobs that appeal to you. 

The agent has taken your name and address and given you 
the rest of the form to fill out. Complete the form so the 


employment center can help you get a job. 


Firth A *e 


Sex: Male Female 


Height. Weight 


Health 


Last grade completed in school 




Kind of work wanted: 




Part-time 


Summer 


Full-time 


Year-round __ 
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DOCUMENT LITERACY, LEVEL 2 (226 TO 275) 



Just over one in five U.S. 
born two-year college 
graduates and about one 
in eight four-year gradu- 
ates can match informa- 
tion in documents that 
contain distractors, and 
they can integrate 
information from several 
places in the document. 
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They can: 

match a single piece of information, with distracting 
information present, or requiring a low level of 
inference. They may also integrate information from 
various parts of the document. 

They are likely to succeed at tasks such as: 

• Locate an intersection on a street map (230) 

• Locate eligibility from table of employee benefits 
(246) 

• Identify and enter background information on 
application for social security card (259) 



EXAMPLE TASK 
What is the gross pay for this year to date? 
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DOCUMENT LITERACY, LEVEL 3 (276 TO 325) 



Over two in five U.S. 
born two-year college 
graduates and over one 
in three four-year 
graduates are at Level 3 
in Document Literacy. 
They can integrate 
several pieces of infor- 
mation and deal with 
rather complex tables 
and graphs that contain 
information not pertain- 
ing to the task. 
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They can: 

integrate several pieces of information from one or 
several documents and deal with rather complex 
tables or graphs containing information that is 
irrelevant or inappropriate to the task. 

They are likely to succeed at tasks such as: 
•Identify information from bar graph depicting 

source of energy and year (277) 
•Use sign out sheet to respond to call about resident 

(298) 

• Enter information given into an automobile main- 
tenance record form (323) 



EXAMPLE TASK 



L 



ERiC 



You need to smooth wood in preparation for sealing 
and plan to buy garnet sandpaper. What type ot 
sandpaper should you buy? 
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Paint Ramoval 
Htavy Stock Removal 
Modiritt Stock Ramoval 
Praparatlon for Soling 
Aftar Sealar 
8ttw«an Coots 
Altar Final Coat 
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Rust and Faint Ramoval 
light Stock Ramoval 
Preparation for Priming 
Finishing and Polishing 
Aftar Pfimar 
Batween Coats 
Aftar Final Coat 
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light Stock Ramoval 
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SAFETY INFORMATION: 

■ Waar approved safety goggles 
when sanding. 




Ultra Flna 



m Ust particte/dust mask or othar 
mains to prevent inhalation of 
sanding dust. 



■ When using power tools, follow 
manufacturer's fecommanded 
procedures and setaty Instructions. 
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DOCUMENT LITERACY, LEVEL 4 (326 TO 375) 



One in four U.S. horn 
two-year college gradu- 
ates and two in five four- 
year graduates are at 
Level 4. They can do 
tasks requiring greater 
inferences and involving 
more detailed informa- 
tion. 
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They can: 

perform tasks that require them to draw higher level 
inferences and numerous responses without being 
told how many are needed. They can also perform 
tasks that contain conditional information. 

They are likely to succeed at tasks such as: 

• Identify the correct percentage meeting specified 
conditions from a table of such information (342) 

• Use a bus schedule to determine appropriate bus 
for given set of conditions (352) (See below) 

• Use a table of information to determine pattern in 
oil exports across years (352) 



EXAMPLE TASK 

On Saturday afternoon, if you miss the 2:35 bus leaving Hancock and Buena Ventura going to 
Flintridge and Academy, how long will you have to wait for the next bus? 

A Until 2:57 p.m. B Until 3:05 p.m. C Until 3:35 p.m. D Until 3:57 p.m. E I don't know 
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TNa but Hna optnrttt Monday through Saturday piovktoig Tboa/aavv*** 
to moat naaghborhooda In tha northaaat atctfon. 
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PM 



12:20 
1:20 
2:20 
2:60 
3:20 
6:60 
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4:60 
5:20 
6:60 
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6:06 
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6:20 
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12:03 
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2:03 
3:03 
9:93 
4:03 
4:99 
5:03 
6:33 
6:03 
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7:03 
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6:46 
6:15 
6:46 
10:16 
11:15 
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2:15 
3:15 
9:46 
4:15 
4:46 
5:15 
6:46 
6:15 
6:45 
7:15 



INBOUND 
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6:46 
7:15 
7:45 
8:15 
6:45 
0:15 
9:46 
10:15 
11:15 
12:15 



1:15 
2:15 
3:16 
3:45 
4:15 
4:46 
5:15 
6:46 
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7:57 
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DOCUMENT LITERACY, LEVEL 5 (376 TO 500) 



Just our in 2.1 U.S. born 
two-year college gradu- 
ates and one in 12 four- 
year graduates are at 
Document Level l.They 
can use complex docu- 
ments that contain 
distracting information 
and can make high level 
inferences. 



All AdujU **3 

2-Yaar Coil*™ fiatfaatea 
Bom ri U.S. -»4 
BomooUJdflU.S. 0 

Bom In US. _a 
Bom outside U.S. -»2 



They can: 

search through complex displays that contain 
several pieces of distracting information, make high 
level inferences from the text, and make use of 
specialized knowledge. 

They are likely to succeed at tasks such as: 

• Use information in a table to complete a graph 
including labeling axes (378) 

• Use a table to compare credit cards; identify the 
two categories used and write two differences 
between them (387) 

• Use a table depicting information about parental 
involvement in a school survey to write paragraph 
summarizing the extent to which parents and 
teachers agree (395) (See below) 

EXAMPLE TASK 

Using the information in the table, write a brief paragraph summarizing 
the extent to which parents and tewhers agreed or disagreed on the 
statements about issues pertaining to parental involvement at their school 



i i i i i i i 

0 20 40 60 
Percenlafle at Level 5 



Parents and Teachers Evaluate Parental 
Involvement at Their School 



Do you agree or disagree that . . . ? 



Level of School 



Total Elementary Junior Hlflh High School 



Our school does a good job of 
encouraging parental involvement in 
sports, arts, and other nonsubject areas 



Parents 
Teachers 

Our school does a good job of 
encouraging parental involvement in 
educational areas 

Parents 
Teachers 

Our school only contacts parents 
when there is a problem with their child 

Parents 
Teachers 

Our school does not give parents the 
opportunity for any meaningful roles 

Parents 
Teachers 



77 
77 



76 
73 



percent agreeing 



74 

77 



79 
85 



73 
80 



55 
23 



22 
8 



82 
84 



46 
18 



18 
8 



71 
78 



62 
22 



22 
12 



64 
70 



63 
33 



28 
7 



Source: The Metropolitan Life Survey ot the American Teacher. 1 987 
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QUANTITATIVE LITERACY, LEVEL 1 (0 TO 225) 



One in 25 U.S. born two- 
year college graduates 
and one in 50 four-year 
graduates are at Quanti- 
tative Literacy Level 1. 
They can perform single, 
relatively simple opera- 
tions. 



All Adults - 22 

2-Year Colleag Graduates 

Bom in U.S. 
8om outside U.S. -*7 



Bom in U.S. *2 
Bom outside U.S. *15 



I I I ) I 1 I 

0 20 40 60 
Percentage at Level 1 



They can: 

perform a single, relatively simple arithmetic opera- 
tion, such as addition. The numbers to be used are 
provided, and the operation to be performed is 
specified.* 

They are likely to succeed at tasks such as: 
• Total a bank deposit entry (191) (See below) 



"However, some were at the lower end of this level where 
they could not perform even these routine tasks. 



EXAMPLE TASK 



You wish to use the automatic teller machine at your 
bank to make a deposit. Figure the total amount of 
the two checks being deposited. Enter the amount 
on the form in the space next to TOTAL, 



Availability of Deposits 

Funds from deposits may not be available for immediate withdrawal. Rease refer to 
your institution's rules governing funds availability for details. 



Creating of deposits and payments is subject to verification *xi collection of actu si amounts 
deposited or paid In accordance with the rules and regulations of your financial institution. 

PLEASE PRINT 



YOUR MAC CARD NUMBER (No P1N$ PLEASE) 

/// 222 3334 

YOUR FINANCIAL INSTITUTION 



YOUR ACCOUNT NUMBER 

987 555 674 



YOUR NAME 



CHECK ONE 



□ DEPOSIT 

or 

□ PAYMENT 



CASH 


$ 


00 


UST CHECKS 
BY BANK NO. 


ENDORSE WITH NAME 
ft ACCOUNT NUMBER 




557 


19 




75 


00 




















TOTAL 







tu 

a 



DO NOT FOLD NO COINS OR PAPER CLIPS PLEASE 
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QUANTITATIVE LITERACY, LEVEL 2 (226 TO 275) 



One in six U.S. born two- 
year college graduates 
*'nd one in 10 four-year 
graduates are at Level 2. 
They can perform an 
arithmetic operation 
using numbers given to 
them or easily located in 
the material. 



All Adults" ,25 

p.YMr college Qttduaias 

Bom in U.S. »17 

Bom outside U.S. 



-.43 



d.Vpar Colteflft Graduates 
BominU.3. — *10 
Bom outside U.S. ——•24 



I I I I 1 I I 

0 20 <0 60 
Percentage at Level 2 



They can: 

perform a single arithmetic operation using numbers 
that are given in the task or easily located in the 
mat°nal. The operation to be performed is either 
described in the task or easily determined from the 
format of the materials (for example, an order form). 

They are likely to succeed at tasks such as: 

• Calculate postage and fees for certified mail (238) 

• Determine the difference in price between tickets 
for two shows (246) (See below) 

• Calculate the total costs of purchase from an order 
form (270) 



EXAMPLE TASK 



The price of one ticket and bus for "Sleuth" costs 
how much less than the price of one ticket and bus 
for "On the Town"? 

THEATER TRIP 

A charter bus will leave from the bus stop (near the Conference Center) 
at 4 p.m., giving you plenty of time for dinner in New York. Return trip 
will start from West 45th Street directly following the plays. Both theaters 
are on West 45th Street. Allow about 1 Vi hours for the return trip. 

Time: 4 p.m., Saturday, November 20 

Price: u On the Town" Ticket and bus $11.00 

"Sleuth" Ticket and bus $8.50 

Limit: Two tickets per person 



BEST COPY AVAILABLE 



QUANTITATIVE LITERACY, LEVEL 3 (276 TO 325) 



EXAMPLE TASK 



Over two in fivo U.S burn 
two -y oar ctollo^v graduates 
and two in Jive four-year 
graduates are at Level 'A. 
They can solve problems 
with two or more numbers 
that must be found in 
printed materia!. 



Suppose that you took the 12:45 p.m. bus from U.A.L.R. Student Union to 
17th and Main on a Saturday. According to the schedule, how many 
minutes is the bus ride? 



All Adults " 



♦31 



2-Yaar Q"flfl* firartnates 

Bom in U.S. •«* 

Bom outtide U.S. » 34 

Bom in U.S. t 40 
Bom otftkto U.S. "21 



1 1 I I I I I 

0 20 40 60 
Percentage at LeveJ 3 

They can: 

perform tasks where two or 
more numbers are typically 
needed to solve the problem, 
and these must be found in the 
material. The operation(s) 
needed can be determined 
from the arithmetic rela- 
tion terms used in the 
question or directive. 

They are likely to 
succeed at tasks such 
as: 

• Use calculator to calcu- 
late differences between 
regular and sale price 
from an advertisement 
(278) 

° Use calculator to deter- 
mine the discount from 
an oil bill if paid within 
10 days (308) 

* Calculate miles per gallon 
using information given 
on mileage record chart 
(321) 



South Highland 








BUS LEAVES 


But arrive* 


Buearrtvee 


BUS ENDS 






from 


at 


tl 


at 






UALR 


20th & 


17th & 


CapKoU 






Student Union 


Woodrow 


Main 


Loui9iana 


WEEKDAYS 1 


A.M. 




559 


551 


950 


959 






6:11 


925 


935 


545 






6*1 


655 


7» 


7:15 






7:11 


725 


756 


7:45 




& 


7*1 




145 


9:15 






8:11 


925 


955 


945 






9:41 


lift 


#55 


9:15 




dx 


9:14 


927 


959 


945 






944 


957 


1059 


10:15 




& 


10:14 


1027 


1059 


10:45 






10:44 


1037 


1199 


li:i5 




& 


11:14 


1127 


1159 


1145 






11:44 


1137 


1259 


1z:io 


P.NL 


6. 


1fc14 


1127 


1259 


1249 






1244 


11*7 


159 


1:11 




& 


1:14 


1*7 


159 


149 






144 


1*7 


259 


2:11 




& 


ft14 


227 


259 


249 






244 


2*7 


159 


l:ii 




& 


9:14 


927 


959 


949 






Ml 


Ml 


1 459 


4:« 




& 


4:11 


4:29 


459 


4:49 






441 


4* 


1 159 


9119 




& 


•:11 


959 


959 


949 






949 


559 


957 


9:17 




6. 


•:11 


922 


959 






& 


949 


997 


759 




1 SATURDAY. J 


A.M. 






551 


9.-00 


659 




-4- 


6:45 


957 


759 


7:15 






7:45 


757 


959 


6:15 




-t- 


9:45 


957 


959 


9:15 






9:45 


957 


1059 


10:15 




-fr- 


10:45 


1057 


1159 


11:15 






11:45 * 


1157 


1259 


12:11 


W*. 


-t- 


1149 


1257 


159 


1:19 






149 


157 


259 


2:19 




-£- 


1*9 


257 


959 


1:19 






949 


957 


459 


4:19 




-*- 


449 


457 


959 


9:19 




-4- 


949 


957 


959 


5:15 






944 


959 


759 
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QUANTITATIVE LITERACY, LEVEL 4 (326 TO 375) 



They can: 

perform two or more operations in sequence or a 
single operation in which the quantities are found in 
different types of displays, or where the operations 
must be inferred from the information given or from 
prior knowledge. 

They are likely to succeed at tasks such as: 
9 Determine correct change using information in a 
menu (331) 

• Use information stated in news article to calculate 
amount of money that should go to raising a child 
(350) 

• Use the eligibility pamphlet to calculate the yearly 
amount a couple would receive for basic supple- 
mental security income (368) 

EXAMPLE TASK 



Estimate the cost per ounce of the creamy peanut 
butter Write your estimate on the line provided. 



Unit price 






You pay 


11.8* peroz. 






1.89 


rich chnky pnt bt 


10693 * 






16 oz. 



Three in 10 U.S. born 
two-year graduates and 
two in five four-year 
graduates are at Level 4 
in Quantitative Literacy. 
They can solve problems 
with two or more num- 
bers that must be found • 
in printed material. 



; AH Adults * 17 

| 9.v f ^ CQlleSfl Graduates 
| Bom in U.S. »30 

| BomoutskfcU.S. ——•15 

Bom in U.S. »40 
Bom outofcte U.S. — *21 
j i I I I I I I 

• 0 20 40 60 

i Percentage at Level 4 



Unit price 


— ^ 




You pay 


1.59 per lb. 






1.99 


creamy pnt butter 


10732 ' 


J 


■ 

WW 


20 oz. 
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QUANTITATIVE LITERACY, LEVEL 5 (376 TO 500) 



One in 20 U.S. born two- 
year college graduates and 
one in 8 four-year gradu- 
ates perform at Quantita- 
tive Literacy Level 5. They 
can handle two or more 
arithmetic operations in 
sequence. 



EXAMPLE TASK 



Aii Adults -*4 

Bom In U.S. -»5 
Bom outside U.S. «1 

4.y««rnnllfly>ftrar*i*itM 

Bom In U.S. »13 

Bom outside U.S. ™*5 
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They can: 

perform multiple opera- 
tions sequentially. They 
can also find the features 
of problems embedded in 
the text or rely on back- 
ground knowledge to 
determine the quantities 
or operations needed. 

They are likely to 
succeed at tasks such 
as: 

• Determine shipping and 
total costs on an order 
form for items in a 
catalog (382) 

• Use information in news 
article to calculate 
difference in time for 
completing a race (405) 

• Use a calculator to 
determine the total cost 
of carpet to cover a room 
(421) 



You need to borrow $10,000. Find the ad for Home 
Equity Loans on page 2 in the newspaper provided. 
Explain to the interviewer how you would compute 
the total amount of interest charges you would pay 
under this loan plan. Please tell the interviewer 
when you are, ready to begin. 



FIXED RATE • FIXED TERM 



14.25% 



HOME 
EQUITY 

I HANC Annual Percentage Rate 
LvAllO Ten Year Term 



SAMPLE MONTHLY REPAYMENT SCHEDULE 
Amount Financed Monthly Payment 

$10,000 $156.77 
$25,000 $391.93 
$40,000 $627.09 

120 Months 14.25% APR 
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Literacy 
and Levels 
of Education 



It is not surprising that on the average, 
the higher the level of one's education, 
the higher the level of literacy (see 
Figure 5). Education results in greater 
literacy, and greater literacy enables one 
to pursue more advanced education. 
Whether the level of literacy seems 
adequate for particular levels of certified 
educational attainment is another ques- 
tion, and we address this in the conclu- 
sion to this section on literateness. 



From Adult 
Literacy in 
America, Irwin 
Kirsch et al., 
Educational 
Testing Service, 
under contract 
with the National 
Center for Educa- 
tion Statistics, 
1993. 
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Prose proficiency tends to rise with education ievei. 
Three-fourths of adults with between zero and eight years 
of school are in Level 1, as are four in 10 high school dropouts. 

Over 75 percent of two-year college graduates and 
85 percent of four-year college graduates reach at least Level 3. 
However, just 4 and 10 percent, respectively, reach Level 5. 



Figure 5: Education Level and Prose Proficiency 



Education 
(Average 
Proficiency) 
0-8 Years 
(177) 

9-12 Years 
(231) 

GED 
(268) 



High School 
Diploma (270) 



Some College 
(294) 

2-Year 
Degree (308) 



4-Year 
Degree (322) 

Grad. Studies/ 
Degree (336) 



Level 1 



* 4 



m 2 



Level 2 



-# 36 



Level 3 



-+ 37 



-#45 



-0 35 



*2 



Level 4 



-+ 32 



1 



Level 5 



_# 47 



10 30 50 70 10 30 50 70 1 0 30 50 70 10 30 50 70 10 30 50 70 

Percentage at Prose Level 
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The 

Distribution 
of Prose 
Proficiency 
by Education 
Level 



Proficiency with prose materials varies 
significantly at each level of educational 
attainment. This has been found to be 
true of achievement in reading and 
mathematics at each grade level as well. 

As shown in Figure 6, the top 25 percent 
of high school graduates score higher 
than the bottom 25 percent of four-year 
college graduates. 

Both admission and graduation require- 
ments vary widely in the U.S. Getting a 
fix on outcomes and making compari- 
sons over time will require recognizing 
this fact. We will want to know about 
the kind of college and the backgrounds 
of the students entering them. This 
survey of literateness does not identify 
the type of college respondents gradu- 
ated from. 
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Special tabulations 
from the National 
Adult Literacy 
Study, 1992. 



Prose proficiency varies significantly at each education level, 
although there is a large overlap in the score distributions. 
For example, the top 25 percent of high school graduates 
score higher than the bottom 25 percent of 
four-year college graduates. 

Figure 6: Percentile Distribution of Prose Literacy Proficiency 
by Education Level, 1992 
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Literacy 
and Weeks 
Worked 



Among all adults there is a steady rise 
in the number of weeks worked during 
the year with increasing literacy profi- 
ciency, whether Prose, Document, or 
Quantitative (see Figure 7). The exten- 
sive analysis of the literacy study has 
shown the many advantages in the labor 
market that go hand-in-hand with 
higher literacy levels. 

This holds true for college graduates as 
well, although the average number of 
weeks worked is fairly flat at the higher 
literacy levels, while it keeps rising after 
Literacy Level 3 in the whole adult 
population. 
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From Adult 
Literacy in 
America, Irwin 
Kirsch et al., 
Educational 
Testing Service, 
under contract 
with the National 
Center for Educa- 
tion Statistics, 
1993. 
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College graduates work more weeks during the year. 
Among college graduates, those at higher literacy levels 

work more weeks. 



Figure 7: Document Literacy and Weeks Worked 





Four- Year College Graduates 
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Literacy 
and Weekly 
Wages 



The higher the literacy level, the higher 
are average wages. This is true for high 
school graduates and those with two- or 
four- year college degrees. 

While wages rise with literacy level, 
Figure 8 also shows that higher wages 
are associated with higher educational 
attainment. At each literacy level, four- 
year graduates have higher average 
wages than two-year graduates, who 
have higher average wages than high 
school graduates. 



From Adult 
Literacy in 
America , Irwin 
Kirsch et al., 
Educational 
Testing Service, 
under contract 
with the National 
Center for Educa- 
tion Statistics, 
1993. 

Some differences 
in average wages, 
from one literacy 
level to the next, 
are not statisti- 
cally significant; 
those with high 
standard errors 
are marked with 
an asterisk. 
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College graduates have higher average weekly wages 
than two-year graduates, who have higher average earnings than 

high school graduates. 
Among college graduates, wages rise wiin literacy levels. 



Figure 8: Document Literacy and Weekly Wages 
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Literacy 
and Age of 
College 
Graduates 



Older college graduates tend to have 
substantially lower Prose Literacy 
proficiencies than do younger graduates 
(see Figure 9). Nine percent of two-year 
graduates, age 55 and over, are at 
Literacy Level 1, as are 8 percent of 
four-year graduates. The average score 
proficiency of this age group is at the 
low end of the Literacy Level 3 range for 
two-year graduates and about the 
middle of this Level for four-year gradu- 
ates. 



From Adult 
Literacy in 
America, Irwin 
Kirsch et al, 
Educational 
Testing Service, 
under contract 
with the National 
Center for Educa- 
tion Statistics, 
1993. 
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While there is little difference in the literacy proficiencies 
of graduates age 16 to 24 and those age 25 to 44, 
graduates age 55 and over have substantially lower literacy levels. 



Figure 9: Age and Prose Proficiency 
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Literacy 
and Age of 
College 
Graduates 

(Continued) 



As described in the prior chart, older 
college graduates have lower literacy 
proficiency levels. Figure 10 shows more 
specifically how age and literacy level 
are related. For two-year graduates, 
there is a drop of 13 years in average 
age from Prose Level 1 to Prose Level 5. 
There is little difference in mean age, 
however, between Levels 3 and 5. The 
age difference for four-year graduates is 
roughly the same. 



From Adult 
Literacy in 
America, Irwin 
Kirsch et ah 
Educational 
Testing Service, 
under contract 
with the National 
Center for Educa- 
tion Statistics, 
1993. 

While average age 
is lower at each 
progression to ;i 
higher level, not 
all differences 
from one progres- 
sion to the next 
are statistically 
significant. 
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Whether with two- or four-year degrees, 
older college graduates have lower average literacy levels than 
younger graduates, with average age generally lower at each 
increment increase in literacy level. 

Figure 10: Literacy and Age of Coilege Graduates 
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Literacy 
and the 
Generations 



As can be seen in Figure 11, the rela- 
tionship between education and literacy 
is intergenerational. 

For respondents at each educational 
level/the higher the education of their 
parents, the higher are their average 
literacy proficiencies. For example, 
respondents with 9-12 years of educa- 
tion, whose parents were college gradu- 
ates, had literacy levels the same as 
high school graduates whose parents 
had less than a ninth-grade education. 

For four-year college graduates, average 
proficiencies ranged from 296 to 324, 
depending on the education level of their 
parents. 



From Adult 
Literacy in 
America, Irwin 
Kirsch et al., 
Educational 
Testing Service, 
under contract 
with the National 
Center for Educa- 
tion Statistics, 
1993. 
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Parent's education greatly influences the literacy levels reached 
by their children. Even among respondents with similar levels 
of education, those with better educated parents 
tend to have higher prose literacy proficiencies*. 



Figure 1 1 : 


Prose Literacy and Parent Education 
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AFTER GRADUATION 



Life after the college degree revolves 
around entering the labor market, con- 
tinuing education, and having a family, or 
some combination of these. While we 
ascribe many desirable purposes to seek- 
ing a college degree, it would be fair to say 
that we expect greatly enhanced prospects 
of doing well in the labor market, landing 
in the more prestigious occupations, 
reducing prospects for unemployment, and 
earning higher salaries. And college 
graduates, we know, do better on all these 
counts. 

We can determine how well college 
graduates are doing a year after receiving 
a bachelor's degree from the surveys con- 
ducted by the National Center for Educa- 
tion Statistics. The last such data are 
available for 1991. For some labor market 
activities, comparable data are also avail- 
able for 1976, 1981, 1985, and 1991, so 
measurement of trends is possible. 

While such trend information helps 
inform us as to how college graduates 
have been faring over the past 15 years, 
business cycles play a large role. Gradu- 
ates leaving school in a recession will not 
do as well, irrespective of the long-term 
trend, and with only a few data points it is 
hard to distinguish long-term trends from 
business cycles. 



The same data permit tracking college 
graduates who enroll in further education, 
and this can be done separately by major. 
It is necessary to keep both outcomes in 
mind — employment and further educa- 
tion — when judging how well students in 
a particular field are doing. For ex^mple, 
biology majors are least likely to be 
employed full-time a year after gradua- 
tion, but they are most likely to be con- 
tinuing their educations. Those in busi- 
ness and management are most likely to 
be employed and least likely to be continu- 
ing their educations. 

This section provides information on: 

• Trends on employment in field of study, 
and on employment in jobs not 
requiring a four-year degree 

• Employment by major 

• Trends in unemployment, full-time 
employment, and further education 

• Further education by major 

• Trends in salary and ratio of female to 
male salaries by major 

• Salary change, by major, from 1976 to 
1991 

• Occupational distribution by major 
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Employment 
in Field of 
Study and in 
Fields Not 
Requiring a 
Four- Year 
Degree 



Since 1985, there has been little change 
in the proportion of recent college gradu- 
ates that have jobs related to their field 
of study, about three in four. These data 
are shown in Figure 12. 

However, more have jobs that do not 
require a four-year college degree, rising 
from 37 percent in 1985 and 36 percent 
in 1987 to 44 percent in 1991. 



Source: National 
Center for Educa- 
tion Statistics, 
Survey of Recent 
College Graduates, 
report for 1991 
survey. 
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About three-fourths of employed Bachelor's degree recipients 

are in jobs related to their field of study, unchanged from 
1985 and 1987. However, 44 percent were in jobs not requiring 
a four-year degree, up from 1985 and 1987. 



Figure 12: Percentage of Employed Graduates in Jobs 
Related to Field of Study, 1985 - 1991 
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Figure 1 3: Percentage of Employed Graduates in Jobs 
Not Requiring a Four- Year Degree, 1 985 - 1 991 
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EniwloVmCIlt * n S enera l> those graduates with degrees 

* * in professional fields are most likely to 

by IVI&joF ^ e employed full-time one year after 

graduation, particularly those with 
business/management and engineering 
degrees (see Figure 14). 

In the arts and sciences, those with 
degrees in the biological sciences are 
least likely to be employed (and, as can 
be seen in Fi r yure 19, the most likely to 
continue their educations). Those most 
likely to work full-time have degrees in 
mathematics, computer sciences, and 
physical sciences. 



Source: National 
Center for Educa- 
tion Statistics, 
Survey of Recent 
College Graduates, 
report for 1991 
survey. 
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Recent degree holders with business management, education, 
and engineering degrees are most likely to be employed 
full-time one year after graduation. 
Biology degree holders are the least likely. 



Figure 14: Percentage of 1989-90 Graduates Employed 
Fuil-Time in 1991 
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Employment 
Relationship 
to Field of 
Study 



There is large variation in the percent- 
age of recent graduates whose jobs are 
related to their field of study, from 95 
percent in the health professions to 53 
percent in the social sciences (see Figure 
15). Of course, those pursuing 
degrees in professional areas have a 
higher expectation of a close relation- 
ship than those with more general edu- 
cation. 



There is also considerable variation 
across the fields of study in the percent- 
age of recent graduates who are 
employed in jobs that do not require a 
four-year degree. The range is from 63 
percent for graduates with a history 
degree, to 19 percent among engineering 
graduates. Of course, those just one year 
out of college have had little time to 
make a connection to jobs with higher 
education requirements, and this will 
change for many as they develop experi- 
ence in and knowledge of the labor 
market. 



Source: National 
Center for Educa- 
tion Statistics, 
Survey of Recent 
College Graduates, 
report for 1991 
survey. 
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Graduates with health, education, and engineering degrees 
are most likely to be employed in jobs related to their 
fields of study. Those with degrees in history, humanities, and 
social sciences are the most likely to be working in jobs 
that do not require a four-year degree. 



Figure 15: Percentage of 1989-90 Graduates Employed Full-Time, 
Who Are Employed in Jobs Related to Field of Study, 
and Jobs Where a 4-Year Degree Is Not Required 
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Employment 
and 

Unemployment 
Trends 



While the unemployment rate for four- 
year college graduates, one year out of 
school, ranged from 3 percent in 1985 to 
5 percent in 1976 (and 4 percent in 
1991), it seems to have mainly reflected 
the business cycle (see Figure 17). The 
rates are lower than the average rates 
for the whole population. 

About three in four college graduates 
are working full-time one year after 
graduation. This is changed little from 
1985 to 1991, although higher than the 
67 percent working full-time in 1976 (a 
recession year) (see Figure 16). Since 
1985, however, considerably more — 
from 23 to 35 percent — are continuing 
their educations (see Figure 18). More 
are working full-time and attending 
school. 



Source: Data on 
full-time employ- 
ment from Digest 
of Education 
Statistics 1993, 
National Center 
for Education 
Statistics, p. 396. 
Other data from 
National Center 
for Education 
Statistics, Survey 
of Recent College 
Graduates, report 
for 1991 survey. 



65 

O 66 



The percentage of bachelor's degree recipients unemployed 
one year after graduation has varied with the business cycle. 
More have been continuing their schooling since the mid 1980s. 



Figure 16: Percentage of Bachelor's Degree Recipients 
Employed Full-Time One Year After Graduation, 1976 - 1991 
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Figure 17: Percentage of Bachelor's Degree Recipients 
Unemployed One Year After Graduation, 1976 - 1991 
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Figure 18: Percentage of Bachelor's Degree Recipients 
Enrolled in Further Education Sometime in Year After 
Graduation, 1976 - 1991 
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Continued 
Education, 
By Major 



Figure 19 shows the proportion of four- 
year college graduates who continue 
their education within a year of graduat- 
ing. It needs to be looked at in relation 
to Figure 14, which shows the percent 
employed. For example, those with 
degrees in biology are least likely to be 
employed full-time, but are most likely 
to be enrolled in further education. 
Business degree holders are most likely 
to be employed and least likely to con- 
tinue their educations in the year after 
graduating. 



Source: Digest of 
Education Statis- 
tics, 1993, National 
Center for Educa- 
tion Statistics, 
p. 393. 
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Recipients of bachelor's degrees in the biological sciences 
are most likely to continue their education and to enroll 
in advanced degree programs. Business and management 
degree holders are least likely to pursue further education. 



Figure 19: Percentage of 1989-90 Bachelor's Degree Recipients 
Pursuing Further Education One Year After Graduation 
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A y\ f*| ll Jl 1 Average salaries, in constant dollars, 

have been fairly flat, for recent four-year 

JSJ JBlljSirV college graduates over the last 15 years, 

^ varying from a low of $21,600 in 1976, to 

a high of $23,300 in 1987, slightly more 
than the 1991 average of $22,700 (see 
Figure 20). 

In the ten-year period from 1981 to 
1991, salaries of female graduates rose 
from 79 percent to 87 percent of male 
salaries. Salaries are close to equal in 
engineering, history, and psychology. 
They are lowest in social sciences, at 83 
percent. These data are shown in 
Figure 21. 



Source: National 
Center for Educa- 
tion Statistics, 
Survey of Recent 
College Graduates, 
report for 1991 
survey. 
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Annua! salaries, one year after graduation, of bachelor's 
degree recipients, have been fairly flat since 1976 
Female salaries are 87 percent of male salaries (up from 79 
percent in 1981), ranging from 83 percent in social sciences 

to 105 percent in history. 



Figure 20: Annual Average Salary of Bachelor's Deqree Recioients 
(Employed Full-Time), One Year After Graduation 1 976 1 99 
in Constant 1QQO n^iior* ' l3/D ia«i, 



in Constant 1990 Dollars 

30 



f -8 

Ctf c 
CO 03 



25 



yjj to on 



Z3 



15 



B21.9 



$22.1 



$23.3 



$21.6 



10 



-r i 1 1 r , 1 r 



1976 1978 1981 1 985 1987 ' ' 1991 

Figure 21: Female Salaries as a Percentage of Male Salaries 
SSBSS )f 1991 (f ° r 1989 " 9 ° bachelor's Degref 

Al1 * • 87 (79% in 1981) 

Professional Fields* 8 7 

Engineering m Q7 

Public Affairs/Soc. Services m 34 w 

Health # 91 

Business/Management* — — # qq 

Education* # 89 



Arts/Sciences Fields* • 85 

History 



Psychology # g6 

Biological Sciences » go 

Math/Computer/Pbys. Sci.* • 9! 

Humanities* • qq 

Social Sciences* • 83 

Other Fields* — # 8 9 



7 5 80 85 90 95 100 
Percentage 



•Statistically significant difference at 95 percent confidence interval 



105 



1 1 1 ' 1 

105 110 



70 



- i n 1976, the highest paid recent four- 
Salary Jt>y year college graduates had engineering 
u/r . degrees, and the lowest paid had educa- 
lVlaj Or, tion an d humanities degrees. By 1991, 

1 0*7 £ "f rh 1 QQ1 the highest paid were in the health 

LViU \>*J At/«/-a. professions, with engineering second; 

education and humanities degree recipi- 
ents were still the lowest paid (see 
Figure 22). 

In 15 years, graduates with degrees in 
physical sciences, mathematics, and 
computer sciences had the greatest gain 
in real wages, 62 percent, followed by 
those with health professions degrees at 
53 percent. The wages of graduates with 
business and management degrees stood 
still, and those of engineering graduates 
rose only 6 percent. 



Source: Digest of 
Education Statis- 
tics, 1993, National 
Center for Educa- 
tion Statistics, 
p. 399. 
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Bachelor's degree recipients (1989-90) with health profession 
degrees had the highest salaries one year after graduation. 
Those with degrees in physical science, computer science, 
and mathematics had the largest percentage increase 

over 15 years. 



Figure 22: Percent Change in Annual Salary of Bachelor's Degree 


Recipients, Employed Full-Time, One Year After Graduation 
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Occupational 
Distribution 
by Major 



Figure 23, shown on the next two pages, 
shows the extent to which those who 
major in a particular field get employ- 
ment in that field one year after gradu- 
ating from a four-year college. They also 
show the other fields in which they find 
employment. 



Health and education majors are most 
likely to be employed in their fields of 
study, followed by engineering and 
math, computer science, and physical 
science majors. There are, of course, 
majors that do not lead specifically to an 
occupational field. For social sciences, 
history, psychology, and humanities 
majors, the largest percentage were 
employed in administrative occupations. 



Source: National 
Center for Educa- 
tion Statistics, 
Survey of Recent 
College Graduates, 
report for 1991 
survey. 
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Health education majors are the most likely to be employed 

in their major field of study. 



Figure 23: Occupational Status of Employed 1989-90 Bachelor's 
Degree Recipients, by Major 
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Continued on next page. 
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Figure 23 (continued) 
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TOWARD HIGHER EDUCATION INDICATORS 



The preceding pages sum up what we 
know, in quantitative terms, about the 
results from, and effects of, higher educa- 
tion. While they provide a worthwhile 
view, they are only small windows into a 
very large house that has many rooms. To 
be sure, there has been a measurement 
movement in higher education, fueled by 
reports in 1986 and gaining wide atten- 
tion, from the Education Commission of 
the States and the National Governor's 
Association's Time For Results. But 
emphasis has been largely on assessments 
used by individual institutions to measure 
progress and improve quality. Quality 
assessment, tailored to instructional needs 
at the classroom level, has been pressed 
consistently for nine years by the Ameri- 
can Association of Higher Education. 
Where states required assessment, they 
left it up to each institution as to how to 
assess, except for Tennessee, whi^h insti- 
tuted a state-wide test. All this has left 
unsatisfied a growing demand for mea- 
surements that would permit generaliza- 
tions about educational quality and 
progress within whole states... or the 
nation. 

Some combination of forces have kept 
alive the question of whether there is a 
compelling need for some indicator system 
that would produce comparable results 
across higher education institutions. This 
was given focus in the national education 
goals set by President Bush and all the 
governors at the historic summit meeting 
in Charlottesville, Virginia in September 
of 1989. They set the objective that by the 
year 2,000: 



The proportion of college graduates 
who demonstrate an advanced 
ability to think critically, communi- 
cate effectively, and solve problems 
will increase substantially. 

In the media attention that surrounded 
the announcement of the goals, and the 
creation of the National Education Goals 
Panel, this goal for higher education got 
little attention, either from the public at 
large or the higher education community. 
The focus was on K-12 education and 
adult literacy. 



A Sample-Based Assessment System 

The Goal 5 Resource Group, estab- . 
lished by the National Education Goals 
Panel in 1990, reported to the Goals Panel 
that to monitor progress on this objective, 
"a new kind of assessment will have to be 
created." The Panel then created a "Tech- 
nical Planning Subgroup" to explore the 
creation of such a measure. The subgroup 
came back to the Panel in June of 1991, 
with the report, "Indicators of General 
Education Outcomes of College Educa- 
tion." This Planning Group urged the 
creation of an assessment and indicator 
system, but also urged that the matter be 
approached with care, and with clear 
understanding of the pitfalls as well as the 
advantages. While believing that there are 
expected general outcomes of a college 
education, beyond specific subject matter, 
the report recognized: 
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• diversity of institutions and major 
programs 

• diversity of views about the purposes of 
higher education 

• diversity of students, in terms of back- 
grounds, age, objectives, and abilities 
as they enter higher education 

The Planning Group recommended the 
creation of an assessment/indicator 
system 

• that assesses a sample of graduates or 
seniors about to graduate, and the 
institutions these students attend 

• that, maintains confidentiality of 
results both with respect to students 
and institutions (however, samples 
could be so designed, or augmented, so 
as to provide information to a partici- 
pating institution for its own use, if it 
is so desired) 

• that draws samples and reports in such 
a way as to provide information by 

the types of the educational 
institution, such as large re- 
search universities, small liberal 
arts colleges, four-year versus 
two-year colleges, etc. 

the characteristics of the course 
of study followed by the students, 
and their majors 

the characteristics of the stu- 
dents in terms of their back- 
grounds, their race/ethnicity, 
gender, whether they are U.S. or 
foreign students, and their skills 
and abilities when they entered 
school from such tests as the 
SAT and ACT 



• that reports achievement and profi- 
ciency in the context of the ability 
levels of the students when they 
enrolled, so that the option exists for 
taking into account these differing 
entering ability levels 

This group examined and rejected 
other approaches, including the develop- 
ment of a standard for achievement, with 
reporting as to how many reached the 
standard. Another Technical Group was 
created later for further development and 
while that group recommended advancing 
on the general lines described above, it 
recommended a standard-setting approach, 
similar to the "levels" now used in the 
National Assessment of Educational 
Progress. We do not believe it is either 
desirable or feasible to establish such 
standards, and we agree that profiling 
proficiency will best serve the purposes of 
measuring progress for the Goals Panel 
and being a constructive force in higher 
education. 

Considerable work was undertaken 
subsequent to this report to the Goals 
Panel. The National Center for Education 
Statistics commissioned a series of papers 
by experts and held intensive workshops 
in 1992.* Out of all this came a request for 
proposals, issued by the U.S. Department 
of Education, to undertake the initial 
development work in the design of a 
sample-based assessment system. How- 
ever, funds for this proved not to be avail- 
able, and an award under the RFP was not 
made. 



*Scc the National Assessment of College Student Learn- 
ing: Identification of the Skills to be Taught, Learned, 
and Assessed, the National Center for Education 
Statistics, August 1994. 
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Indirect Measures of Outcomes 

The Planning Group recognized the 
enormity of getting into place a sample- 
based assessment system. It also recom- 
mended that work proceed on indirect 
indicators "that could be collected from 
records now available or added to surveys, 
and that could serve to encourage educa- 
tional progress in higher education." We 
report here of subsequent work under- 
taken by Peter Ewell and his colleagues, 
funded by the National Center for Educa- 
tion Statistics, to identify indirect mea- 
sures of cognitive outcomes. We also report 
on an effort by the Community College 
Roundtable to identify indicators of the 
core mission of community colleges (which 
are not specifically directed toward cogni- 
tive outcomes). 



Instructional "Good Practice" Indica- 
tors in Higher Education 

Solid work has now been undertaken 
that could undergird a system of indirect 
indicators — proxy measures of general 
educational achievement.* They result 
from examining a large number of studies 
of the cognitive results of institutional and 
student practices and behaviors. Indica- 
tors of good practice were investigated in 
the following areas: 

1. Institutional Requirements, such 
as graduation requirements or 



*A Preliminary Study of the Feasibility and Utility for 
National Policy of Instructional "Good Practice" Indica- 
tors in Undergraduate Education, prepared for the 
National Center for Education Statistics, by Peter T. 
Ewell, assisted by Cheryl P. Lowell, Paula Dressier, and 
Dennis P. Jones, of the staff of the National Center for 
Higher Education Management Systems, August, 1.994. 



explicit demonstrations of com- 
petencies in subjects, having to 
write a major research paper, or 
a "capstone" experience such as 
an internship or senior thesis. 

2. Instructional "Good Practices," 
such as "active learning," low 
class size, frequency of writing or 
speaking, frequency of contact 
with faculty members, or fre- 
quency of tutoring other stu- 
dents. 

3. Student Behaviors and Self- 
Reported Gains, such as indepen- 
dent research, student reports of 
gains they have made in cogni- 
tive areas, reports of finding 
classwork challenging. 

These areas, illustrated above by a few 
examples, served as a framework for the 
search for good practice indicators through 
a two-staged approach. First, there was a 
general search through research studies to 
identify promising ones, and second, there 
was an in-depth analysis of the studies 
identified. The pages that follow show the 
results of this ambitious and extensive 
effort in summary form. The summaries 
provided by the authors also describe the 
"available methods for collecting indica- 
tors data" for each of the indicators listed. 
For these, we refer the reader to the origi- 
nal report. The report also provides a 
bibliography of all the research studies 
used or consulted. 



Indicators of Effectiveness of 
Community Colleges 

Another effort at identifying indicators 
of effectiveness was undertaken by the 



ERLC 



78 



79 



SUMMARY CHART OF POTENTIAL "GOOD PRACTICE" INDICATORS 



Indicator Domain/ 
Dimension 


Relative Strength of 

Association with 
Goal 5.5 Outcomes 


Available Methods 
for Collecting 
Indicators Data 


Relative 
Ease of 
Data* 
Gathering 


Policy 
Relevance 


Overall 
Potential 


A. Institutional Requirements: 


Broad General Education 
Curriculum/Requirements 


Weak/Moderate 


- Catalogue Review 

- Institutional Questionnaires/ 
Inventories (e.g., Peterson 1S37) 


Difficult 


High 


Low 


Breadth of Coursetaking/ 
Types of Coursetaking 


Moderate 


•"Breadth" and "Depth" of 
Courses Taken (Zemsky 1 989) 
- "Differential Coursework 

Methodology" (Ratcliff & 

Associates 1988) 


Moderately 
Difficult 


Moderate 


Moderate 


Special Courses/ 
Instructional Designs 


Moderate (but likely 
derivative) 


- Catalogue Review 

• Institutional Questionnaires/ 

Inventories (E.G., Gamson & 

Poulsen 1989) 


Difficult 


Low 


Low 


Levels of Expectation 


Moderate/Strong (via 
association with "Small 
Liberal Arts College" effect) 


- rating examinations and course 
materials by level of difficulty 
(e.g., Braxton & Nordvall 1985) 


Difficult 


Moderate 


Moderate 


B. Instructional "Good Practice": 


Class Size and Structure 


Low/Moderate (but likely 
derivative) 


- Institutional Survey/Statistics 


Moderately 
Difficult 


High 


Moderate 


"Active Learning'* practices in 
class 

- Practice of Skills 

- Frequent Feedback 

- Peer Interaction 


Strong 
Strong 
Strong 


- Faculty Surveys 
e.g., 7 Principles Survey 
(Gamson & Poulsen 1989), 

UCLA Faculty Survey 

• Student Surveys 
e.g., CBEQ (Pace 1987) 
CIRP (Astin & Associates 1992) 


Not difficult 


Moderate 


High 


Wider Institutional Environment 

- Invoivement 

- Student/Faculty Contact 


Strong/moderate (complex 
interaction of factors) 

Strong/moderate (but also 
may require data on nature 
of interaction) 


-CSEQ (Pace 1987) 

- CIRP (Astin & Associates 1 992) 


Not difficult 
Not difficult 


Moderate 
Moderate 


High 
High 


C. Student Behavior: 




Time on Task 


Strong/Moderate (but also 
requires quality of 
investment measures) 


-CSEQ (Pace 1987) 

- CIRP (Astin & Associates 1992) 


Not difficult 


Moderate 


High 


"Quality of Effort"/ Involvement 
and Investment 


Strong 


•CSEQ (Pace 1987) 


Not difficult 


Moderate 


High 


D. Self-Reported Cognitive Development: 




Moderate/Strong 


-CSEQ (Pace 1987) 

- CIRP (Astin & Associates 1 992) 

- ACT-ESS (ACT 1982) 

- NCHEMS SOIS (NCHEMS 1983) 


Not difficult 


N/A 


Moderate 


Source- A Preliminary Study of tho Feasibility and Uti'ity for National Policy of Instructional 'Good Practice" tnd:cator$ in Undergraduate Education , pre pa red for the National 
Center (or Education Statistics, U S. Department of Education, p 36-38 
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Community College Roundtable/ The 
principal purpose of the Roundtable's 
report is to suggest a set of indicators an 
individual college could use to gauge its 
effectiveness. It is conceivable, however, 
that widespread use and adoption could 
permit aggregation to a set of colleges, or 
to whole states. The indicators deal with 
"the major mission tasks that a commu- 
nity college must accomplish to be success- 
ful." They do not deal with all aspects, nor 
address specific student learning 
outcomes. 

A very brief summary of the indicators 
is provided below. 

Student Progress: 
Student Goal Attainment 
Persistence (Fall to Fall) 
Degree Completion Rates 

Career Preparation: 

Placement Rate in the Workplace 
Employer Assessment of Students 



Transfer Preparation: 

Number and Rate Who Transfer 
Performance After Transfer 

Developmental Education: 

Success in Subsequent, Related 
Coursework 

General Education: 

Demonstration of Critical Literacy Skills 
Demonstration of Citizenship Skills 

Customized Education: 

Client Assessment of Programs and 
Services 

Community Development: 

Assessing Responsiveness to 

Community Needs 
Participation Rate in Service Area 

The report provides a measure for each 
indicator, the data sources, and additional 
or alternative measures. 



'Community Colleges: Core Indicators of Effectiveness, a 
report c~the Community College Roundtable, published 
by the American Association of Community and Junior 
Colleges, 1994, The quote is from the Preface, written by 
Richard Alfred, Paul Kreider, and Kay McClenney, 
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CONCLUSION 



As we said at the outset, these data 
provide only a limited view of certain 
aspects of higher education. The data are 
solid, however, particularly the results 
that derive from tests, the literacy assess- 
ment, and statistical series. We can have 
confidence in what they do tell us. 

With regard to the synthesis of small- 
scale research studies, there is strong 
evidence that college has a positive effect 
on verbal and quantitative skills. Also, 
there is moderate to strong evidence of a 
positive effect on critical thinking, although 
the magnitude is unclear. There is also 
moderate evidence of a positive effect on 
oral and written communication and 
general intellectual and analytical skill 
development, although the magnitude of 
the effect is also unclear. 

Quantitative and Analytical scores rose 
on the Graduate Records Examination 
from 1981 to 1993, with the volume of test 
takers up almost by half (Verbal scores 
were stable). In the same period, scores on 
subject matter tests rose in eight subjects 
and fell in seven. In four of the tests where 
scores increased, the number of test takers 
also increased. Scores have risen on the 
Graduate Management Admissions Test 
as well. 

Three-fourths of two- and four-year 
college graduates (born in the U.S. or its 
territories) scored in the third and fourth 
highest levels of literate ness on a national 
assessment given to a household sample of 
all adults age 16 and over. We ask the 
reader to look at these literacy levels in 
terms of what adults at those levels can 
do, and make their judgments about 



whether the performance of college gradu- 
ates meets their expectations. While these 
skill levels are not negligible, neither do 
they seem to the authors to be impressive. 
And performance is lower for the older 
college graduate population. At least in 
the area of literateness, as this assess- 
ment measures it, it is hard to disagree 
with what Alexis DeToqueville said long 
ago, that "A middling standard is fixed in 
America for human knowledge. All 
approach as near to it as they can, some as 
they ascend, others as they descend." But 
we don't know why younger graduates 
perform better than older ones. 

If you judge Level 3 to be below what 
you would expect for four-year college 
graduates, then it is important to note 
that just over half are at Level 4 or 5.* 
Only 11 percent are at Level 5. 

Leaving out the effects of aging, 8 
percent of four-year college graduates 
under age 25 are at Levels 1 and 2 (mostly 
at Level 2) as are 18 percent of two-year 
graduates (also, mostly in Level 2). Few 
would disagree after reading page 29 that 
the Prose capabilities of people at this 
level are fairly low. 

We must remind ourselves that these 
data have only limited direct relationship 
to the objectives of higher education. 
Colleges and universities do not have 
courses based on this assessment of 
literateness. Most probably assume lit- 
eracy or their definition of it as an 

essential characteristic of their applicants. 



'Born in USA or its territories. 
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Universities were established to form 
preachers of the gospel, to train civil 
servants, to prepare teachers, and to hone 
the skills of engineers and surgeons. The 
early theologians, philosophers, and seek- 
ers of truth have become today's research- 
ers and scholars. 

Nevertheless, we suspect that if you 
ask the man or the woman on the street 
whether people who graduated from col- 
lege should be able to do the tasks in, say 
Levels 4 and 5, they would likely answer 
yes. While these are the tasks people 
encounter in daily life, many involve 
problem solving; the context for such tasks 
should not be critical, as these skills are 
supposed to be transferrable from one 
context to another. However, literacy 
assessment is not designed to be an 
assessment of higher education outcomes, 
and any measures used directly for that 
purpose should be developed through a 
process of purposeful development and 
consensus building. 

We hold out the expectation that a 
college degree will result in success in the 
employment world. Indeed, higher educa- 
tion is the only well established route for 
the transition from school to work in the 
United States. Of course, college gradu- 
ates, on average, do better financially than 
those without degrees; high school gradu- 
ates have been losing ground in terms of 
real wages in the last couple of decades. 
Annual salaries of four-year graduates 
(one year after graduation) were just a 
little higher, in constant dollars, in 1991 
than in 1976, and a little lower than in 
1978; yet the cost of getting that college 
degree has risen sharply since 1976. 

The percent of graduates employed 
full-time a year after graduation has risen 
from 67 percent in 1976 (a recession year) 



to 74 percent in 1987 and 1991, at the 
same time that the percent enrolled in 
further education has grown from 27 
percent in 1976 to 35 percent in 1991 
(after dipping to 23 percent in 1985) 
Many of these are part-time enrollments. 

The percent of graduates who are 
employed a year later in jobs not requiring 
a four-year degree rose from 37 percent in 
1985 tc 44 percent in 1991, although the 
percent in jobs related to their field of 
study has held steady at over 75 percent. 
Over long stretches there has been a trend 
in which during each decade more college 
graduates take jobs that were held by 
those with less than a college degree the 
prior decade. Thus, the perception of 
whether a job requires a college degree 
changes over time, and the statistics 
reflect these shifting perceptions, as they 
come from the graduates themselves. 

Not unexpectedly, salaries vary consid- 
erably for recent college graduates, as does 
the change in salaries over time. Those 
with degrees in the health professions 
were the highest paid, earning an average 
of $31,500 in 1991. This is 53 percent 
higher (in constant dollars) than 1976. 
They surpassed engineers, at $30,900, 
whose wages grew by just 6 percent. The 
lowest paid were those with degrees in 
education and the humanities, at $19,100, 
and growth rates of 27 percent and 37 
percent, respectively. Those with degrees 
in business and management increased 
only 1 percent. The largest growth was 62 
percent for those with degrees in the 
physical sciences, mathematics, and com- 
puters. Market demand shifts, and salary 
levels adjust accordingly. 

* * * 
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No sweeping statements about the 
state of higher education, or whether 
quality is improving overall or not, are 
possible from the data available. Some 
useful knowledge about particular aspects 
is available. Do we need to know more? 

The demand for better information will 
likely get stronger, as finances are con- 
strained, costs rise with inflation in the 
higher education sector, and concern for 
economic competitiveness mounts. Also, 
the education reform movement keeps the 
spotlight on quality... and a quality move- 
ment... Total Quality Management... 
spreads throughout the economy The 
nation's governors, the President, and now 
the U.S. Congress, have set a goal for the 
year 2000 for higher education, for which 
progress cannot now be measured. 

Good quantitative information will 
help all involved make better choices. But, 
as discussed earlier in this report, quanti- ' 
tative information alone is insufficient for 
judging whether higher education is ful- 
filling all of its many stated purposes. 
Efforts recently begun show promise of a 
thoughtful and constructive beginning. 
With enough resolve, full implementation 
could continue: 

• A follow through on creating a sample- 
based assessment to produce indicators 



of critical thinking, problem solving, 
and effective communication. The effort 
of the National Center for Education 
Statistics was aborted because of lack 
of funding. 

• A parallel effort, creating indicators of 
good practice, based on solid research 
on the relationship of such practices to 
cognitive outcomes. This work has been 
carried a long way by Peter Ewell and 
his colleagues. Also, the Community 
College Roundtable has agreed on 
indicators of effectiveness for commu- 
nity colleges. 

America may now have the best higher 
education system in the world. The ques- 
tion though is whether it is as good as it 
can be and as good as we want it to be, 
and whether it is realizing the potential 
embedded in the vast resources devoted to 
it. The commitment to measure results 
and the commitment to act are likely 
interwoven. A narrow-minded view of 
what constitutes good information would 
not reflect critical thinking in a nation 
dedicated to the broadening of minds 
through liberal education. Properly devel- 
oped and sensibly used, however, a good 
system of indicators can help higher edu- 
cation improve individual opportunity and 
well being, as well as make the U.S. a 
stronger nation. 
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